How does parfor divide the input array?

Hi Guys

I was wondering how

'parfor i=1:length(data)'

is divided between workers?! what are the starting indexes for each workers. Assume that we have 8 workers.

Answer 1

[Edric Ellis]

Walter's answer is conceptually correct, but as he observes, misses some of the nuances of the implementation. The current implementation sends blocks of iterations to the workers, starting at the end of the range and running backwards. It sends blocks of decreasing size to attempt to achieve good load-balancing. It's relatively easy to deduce this in practice by having each worker print out the iterations it is working on, and the result of getCurrentTask().

If you want more control over what work is assigned to each worker, use parfeval.

Answer 2

[Walter Roberson]

Dividing the range up with starting indices for each worker is what would happen with distributed arrays, not with parfor itself.

parfor itself creates a queue of individual index values, queued in an unspecified order (but commonly the first entry is the highest index of a parfor loop that is in ascending order.) When a worker finishes with one value and reports back, parfor stores the results, and then gives the next available item in the queue to the worker; which item is next is going to depend on exactly what order the workers finished and managed to interrupt across cores. The lower the overhead that parfor has, the more chaotic the exact assignment of indices to workers becomes. But the order is also affected by the details of how the hardware multi-core synchronization happens when there are multiple cores queued to interrupt.

I might have some details wrong about how parfor is implemented in practice in current systems, but I think I have stayed within the bounds of the interface "contract" -- what the interface is allowed to do if it wanted to.

Slicing the range by workers is in the "contract" for distributed arrays but not for "parfor".