How is the work distributed in parfor loop?

In a scenario with a parfor loop consisting of 10000 iterations distributed over 10 workers, is each worker initially assigned 10000/10=1000 tasks at the beginning of the parfor loop, or does each worker receive one task initially and subsequently get assigned new tasks upon completion?

Specifically, I am encountering a situation where approximately 500 iterations of a parfor loop (of total 10000 iterations) are computationally heavy while the remaining ~9500 iterations are computationally lighter. When running this parfor loop, I notice that the execution is initially extremely fast, with around 90% completion within 5 minute. However, the remaining 10% takes significantly longer, up to 20 hours. My suspicion is that the computationally heavy iterations are being assigned to a single worker, which handles them sequentially.

Answer 1

[Edric Ellis]

If you know in advance which iterations are the time-consuming ones, then you could consider running those together as a separate parfor loop, and the default iteration partitioning described by @Walter Roberson will probably do a decent job of keeping all workers busy.

If you do not know in advance, then there are two strategies that you could use, and maybe you could use both together:

1. Shuffle the order of the iterations using randperm or similar to try and end up with the long-running iterations in different sub-ranges.
2. Use parforOptions to force small sub-ranges.

[Luqman Saleem]

Thank you. I have a general idea of which iterations are computationally intensive. My current parfor loop structure is as follows:

parfor ix = 1:10000
    % code
end

Specifically, I have identified the iterations from ix = 2000 to 2500 and from ix = 7000 to 7500 as computationally heavy. Could you please provide a sample code illustrating how I can instruct the parfor loop to distribute these iterations among multiple workers?

[Walter Roberson]

For example, set the RangePartitionMethod to [50*ones(1,(1950-0)/50), 20*ones(1,(2500-2000)/50), 50*ones(1,(7000-2500)/50)), 20*ones(1,(7500-7000)/20), 50*ones(1,(10000-7500)/50)]

or something like that.

Answer 2

[Walter Roberson]

You can control how parfor divides up the iterations by using parforOptions() RangePartitionMethod https://www.mathworks.com/help/parallel-computing/parforoptions.html#mw_5eb7f106-9fa3-45da-a7ae-6f5b0db4bef0

The default is "auto", which divides most iterations up into chunks, then divides most of the remaining iterations into smaller chunks, then hands out the remaining iterations one at a time.

[Luqman Saleem]

Got it. Thanks.