Some workers go idle in parfor creating a waste of computation time

14 ビュー (過去 30 日間)
H R
H R 2020 年 4 月 5 日
回答済み: Edric Ellis 2020 年 4 月 6 日
Hi all,
I have a for loop of 90 iterations (array xx has 90 rows and say 20 colums - each row is an iteration). Each iteration evaluate a costly function. the evaluation time for some functions is around ~ 24 hours while for some others is around 10 hours.
I am using parfor with a pool of 20 workers. However, I observe that some of the workers go idel after finishing the iteration faster. this can create a large waster of time. How can I request parfor worker to immediately run a new iteration after it finishes its current evlauation?
myCluster = parcluster('local');
myCluster.NumWorkers = 20;
saveProfile(myCluster);
p=parpool(20, 'IdleTimeout', Inf);
opts = parforOptions(p, 'RangePartitionMethod', 'fixed', 'SubrangeSize', 1);
parfor (k=1:size(xx,1),opts)
out(k,:)=f(xx(k,:), k);
end

採用された回答

Edric Ellis
Edric Ellis 2020 年 4 月 6 日
The parfor implementation uses a number of different strategies to try to keep the workers busy. However, sometimes these might turn out not to work well for a particular problem. There is no way to explicitly request the workers start the next iteration - that is all handled automatically by the parfor implementation. I'm a bit surprised that you're seeing workers being idle (for a noticeable amount of time) and then later becoming busy. (Of course, at the end of the loop, there's always going to be a "tail" when each worker has finished its work). I wouldn't normally expect that to happen unless you've got a huge amount of data to transfer (and even then, the parfor implementation tries pretty hard to hide data transfers by running them concurrently with other stuff).
One alternative you might explore is to use parfeval rather than parfor. This uses a simpler scheduling mechanism than parfor, but it might work better for your situation. Briefly, each call to parfeval schedules a single remote function evaluation on a worker. You call fetchNext or fetchOutputs to collect the results. A bit like this:
% schedule work
for k = 1:size(xx,1)
f(k) = parfeval(f, 1, xx(k,:), k); % 1 is the number of outputs you're requesting
end
% Collect results
for idx = 1:size(xx, 1)
[k, data] = fetchNext(f);
out(k,:) = data;
end
There's an introduction to parfeval here in the doc.

その他の回答 (0 件)

カテゴリ

Help Center および File ExchangeParallel for-Loops (parfor) についてさらに検索

タグ

製品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by