optimize cluster configuration for virtual machine

1 ビュー (過去 30 日間)
Roy 2022 年 1 月 29 日
コメント済み: Raymond Norris 2022 年 1 月 31 日
I'm testing a parfor loop where each parallel iteration has to do some pretty heavy processing (~70 GB peak mem usage, ~3 h runtime without parfor). Eventually I need to run this on hundreds of datasets, so I want to ensure a reasonable cluster configuration. I'm running Matlab 2020a (Win 10) interactively on a configurable virtual machine (256 GB RAM with 32 virtual processors).
Here's what I do:
myClus=parcluster('local'); %from virtual machine perspective, all hardware is local
myClus.NumWorkers=3; %limited by RAM: 3*70 GB < 256 GB capacity
myClus.NumThreads=10; %10 threads (=virtual processors?) per worker: 3*10 < 32 available processors
%parfor loop...
Is this a reasonable (or at least not terribly inefficient) setup in this situation? Specifically:
  • is setting NumThreads = #virtual processors/NumWorkers (as I'm essentially doing) an acceptable strategy? I'm assuming a thread is equivalent to a virtual processor in my case, but I'm not entirely sure
  • would increasing NumThreads be expected to lead to further improvements? Since I have the luxury of configuring my hardware, I could conceivably go for e.g. 64 processors and set NumThreads=20
  • bonus points for other optimization suggestions

回答 (1 件)

Raymond Norris
Raymond Norris 2022 年 1 月 31 日
When you run the code serially, do you see high core activity? If not, you might not see any appreciable difference increasing NumThreads. Depends on the work the parfor loops is running. Might be worth testing it by throttling NumThreads and seeing what performance improvements you get.
Have you consider at some point the need to scale your code to a cluster/cloud?
  2 件のコメント
Raymond Norris
Raymond Norris 2022 年 1 月 31 日
I don't have a rule of thumb, but keep in mind, these are threads, not processes. By default, MATLAB has already been mapping the computation thread count to your physical core count all this time. If you haven't seen a degregation in your system, but fine to leave it as is. But as you say, testing might be the best confirmation.


Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by