Parfeval blocks parfor loop evaluation

The existance of any parfeval job blocks the execution of a parfor loop. This occurs even when available pool workers are available, and should be free to execute the parfor loop.

On my machine, with 32 cores, running this:

function testParallelBlocking
nIter = 3;

pool = gcp;

j = parfeval(pool,@backgroundFcn,1);

for i = 1:nIter
    forLoopTime(i) = now;
end

parfor i = 1:nIter
    parforLoopTime(i) = now;
end

disp(['Parfeval completed at ' datestr(j.fetchOutputs,'HH:MM:SS:FFF')]);

for i = 1:nIter
    disp(['For loop completed at ' datestr(forLoopTime(nIter),'HH:MM:SS:FFF')]);
end

for i = 1:nIter
    disp(['Parfor loop completed at ' datestr(parforLoopTime(nIter),'HH:MM:SS:FFF')]);
end

end

function out = backgroundFcn

pause(5);

out = now;

end

produces this output:

Parfeval completed at 15:15:25:244
For loop completed at 15:15:20:232
For loop completed at 15:15:20:232
For loop completed at 15:15:20:232
Parfor loop completed at 15:15:25:269
Parfor loop completed at 15:15:25:269
Parfor loop completed at 15:15:25:269

showing that while the rest of the code continues to run, the parfor loop does not begin until the parfeval job has completed.

Is there any way to have the parfor loop execute while the parfeval job is also running in the background? I have tried limiting the number of parfeval workers by changing

parfor i = 1:nIter

to

parfor (i = 1:nIter,nWorkers)

where nWorkers < (number of cores - number of pending parfeval jobs) but it does not change the behavior.

Thanks for your help.

Answer 1

[Edric Ellis]

This is expected behaviour. Unfortunately, there is currently no way to overlap parfeval and parfor. Your best bet is probably to adapt your parfor loop to use parfeval instead (I appreciate that this might not be terribly convenient).

[Michael]

Edric,

Thank you for your response - yes, it is not really feasable for me to rewrite all parfor loops as parfeval loops.

Is this something that is on the Mathworks timeline for a fix?

Is there a way to check if any parfeval tasks are executing? If I set the numWorkers argument to the parfor loop to 0, the loop will run as a simple "for" loop and not get blocked by parfeval.

What I'd like to do is something like:

pool = gcp;
if PENDING_PARFEVAL_TASKS
    nWorkers = 0;
else
    nWorkers = pool.NumWorkers;
end

parfor (i=1:N,nWorkers)
    .
    .
    .
end

The lack of interoperability of parfeval and parfor really limits the functionality of parfeval.

Thanks,

Michael