Codistributed arrays taking too long to run

Hello fellow programmers.

I've been exploring the distributed arrays functionality on Matlab. I tried doing a simple matrix multiplication, where both matrices have the same dimension, and comparing the time it takes the serial and parallel execution to run.

Here's the code, not "parallelized" (is that a word?)

N = 5000;
A = eye(N);
B = magic(N);

tic
C = A*B;
toc

I try to run it in parallel by distributing the array of the matrices between the 4 workers I have available.

N = 5000;
A = eye(N);
B = magic(N);

spmd
    A = codistributed(A,codistributor1d());
    B = codistributed(B,codistributor1d());
end

tic
spmd
    C = A*B;
end
toc

However when I run this code it takes a lot of time to distribute the matrices between the workers (with the codistributed() function) and the matrix multiplication takes significantly longer (about 5/6 times longer).

If anyone could tell me what I'm doing wrong I'd appreciate. Am I not understanding how distributed arrays should be used?

Cheers Daniel

Answer 1

[Jill Reese]

Here is an example of how to properly benchmark an operation on distributed arrays:

http://www.mathworks.com/help/distcomp/examples/benchmarking-a-b.html

You are comparing the implicit parallelism (multithreading) of core MATLAB to the explicit parallelism (MPI communication between MATLAB workers that are themselves running singlethreaded) of the Parallel Computing Toolbox distributed arrays. If the input matrices A and B fit in memory on a single machine with enough memory left over to perform the operation and store the output, it's often pretty hard to beat the performance of the multithreaded operation. Distributed arrays provide a clear benefit when the problem exceeds the memory available on a single machine.

If you set up a benchmark for mtimes (*) following the example that I've linked, you can investigate the performance you can expect from your own machine. I think that the codistributor you have chosen is not unreasonable for this operation.

Answer 2

[José-Luis]

It's because even though you distribute your array amongst the workers, matrix multiplication might not be efficient, depending on how you distribute the array. In order to compute it you will need to access data in the other arrays, considerably slowing your code due to communication overhead.