Processing blocks of data in parallel

I have a data set which is 4 million rows by 80 columns wide. I want to split the data up into equal sized blocks so that each block is processsed in Parallel.

Would the Parallel computing tool box allow a function to be written such that each block is processed in parallel?

Answer 1

[Edric Ellis]

Your best bet is to try and restructure your code to use a PARFOR loop. For example, you could do something like this:

N = 4e6;
indata = rand(N, 80);
outdata = zeros(N, 1); % results
parfor ii = 1:N
  outdata(ii) = myfcn(indata(N,:));
end

You can run PARFOR loops in standard MATLAB to check they work, but you need Parallel Computing Toolbox to run in parallel.

Another option is to use distributed arrays - this lets you split your input data across the memory of multiple machines (you need MATLAB Distributed Computing Server licences to run workers on multiple machines). That way, you never need the whole of your large data set in a single machine's memory.

Distributed arrays are built on top of MPI - Parallel Computing Toolbox ships with a build of MPICH2.

[adam cox]

Great answer - thanks Edric

The reference to the function from my other question needed to generate random numbers of -1, 0 or 1 [and I used this function: round(low + (high-low) * rand);]

How would this change your suggested code example?

Thanks in advance.

[Walter Roberson]

Please see

http://www.mathworks.com/help/toolbox/distcomp/distributed.rand.html

http://www.mathworks.com/help/toolbox/distcomp/codistributed.rand.html

http://www.mathworks.com/help/toolbox/distcomp/bsic4fr-1.html

Answer 2

[Daniel Shub]

This depends on the processing. If each block can be processed independently from the other blocks, then yes the Parallel computing toolbox can handle that. If the processing for each block depends on the other blocks, I don't think the Parallel computing toolbox has a simple MPI (although I have never tried).

With standard MATLAB you can see if parfor can help.