parfor loops with large outputs

Hi,

I'm using MATLAB's parallel toolbox on a shiny server (32 physical cores, 64 logical, 512GB RAM). All my parallelization needs can be classified as 'embarrassingly parallel', e.g.:

          parfor i_par = 1:n_tot
              [CalcOutputs_array(:,:,i_par), AdvancedCalcOutputs_array(i_par)] = CPU_DemandingFunction(InputsStruct_array(i_par));
          end

In order to minimize overheads, i've already split the input such that each worker receives only what it needs.

The output, however, can become quite large (1GB-10GB), and data collection from the workers is vastly over-weighting the calculation itself.

I'm positive that this is the case since by lowering the amount of data outputted from the function (i have a flag that controls it) - the runtime improvement factor approaches the # of cores i'm using.

Can that be improved? Is there any remedy for data-collection overheads?

Thanks a lot!

Yanir

[José-Luis]

This is a very general question.

You could:

1. Suppress the output arguments you don't need
2. Save to a buffer and save that to disk when it becomes too large.
3. Have each worker save its own output, not aggregating. You can put all the files together yourself afterwards
4. Use sparse arrays

etc...