use parfor if larger than certian size

Is there an easy method to specify in the parfor loop to use parfor only if a value is larger than a certian size? I.E. I have a user selecteable option, and if they only chose to select to perform the operation on a few files, the time it takes to initiate the parllel pool is longer than it would take to perform a normal for loop. But if the user selects a large number of files, the parfor loop is significantly faster even if one has to wait for the parallel pool to start.

Answer 1

[Raymond Norris]

Hi Shane,

There might be several options here, in no particular order

1. For starters, you might consider starting the pool (would it be local?) as part of the startup process of your application. By default, the parallel pool will shutdown after a timeout period. You might want to suggest they disable this, so that it doesn't shutdown in case they don't use it for a while. Of course this may have some performance degregation if you don't need the pool running at all. When your application is done, and if you've had them override auto-shutdown of the pool, you might want to close the pool as well.

2. In R2020a we introduced "threads" pool, which might be quicker to start. You can read more here

https://www.mathworks.com/help/parallel-computing/choose-between-thread-based-and-process-based-environments.html

p = parpool("threads");
parfor fidx = 1:numel(files)
    process(files{fidx})
end

3. parfor takes several arguments. Two to look at are:

• Specifiying the maximum number of workers to use. For example

if nfiles>=threashold
    psize = 4;
else
    % Not enough files to process in parallel
    psize = 0;
end
parfor (fidx = 1:numel(files),psize)
    process(files{fidx})
end

If psize is 0, then parfor operates as a for (though maybe a bitter slower? worth checking).

• Request a cluster pool (introduced in R2019a), rather than a parallel pool. By specifying a parfor options, jobs are submitted via batch submission. There's more to understand about cluster pools, so read more here

https://www.mathworks.com/help/parallel-computing/parforoptions.html

The advantage to this is that it doesn't take as long to start the pool. This is more applicable to jobs running on an cluster.

cluster = parcluster;
opts = parforOptions(cluster);
parfor (fidx = 1:numel(files),opts)
    process(files{fidx})
end

Raymond

[Shane Sullivan]

Thanks Ryan, this info helps.