optimize cluster configuration for virtual machine
1 ビュー (過去 30 日間)
I'm testing a parfor loop where each parallel iteration has to do some pretty heavy processing (~70 GB peak mem usage, ~3 h runtime without parfor). Eventually I need to run this on hundreds of datasets, so I want to ensure a reasonable cluster configuration. I'm running Matlab 2020a (Win 10) interactively on a configurable virtual machine (256 GB RAM with 32 virtual processors).
Here's what I do:
myClus=parcluster('local'); %from virtual machine perspective, all hardware is local
myClus.NumWorkers=3; %limited by RAM: 3*70 GB < 256 GB capacity
myClus.NumThreads=10; %10 threads (=virtual processors?) per worker: 3*10 < 32 available processors
Is this a reasonable (or at least not terribly inefficient) setup in this situation? Specifically:
- is setting NumThreads = #virtual processors/NumWorkers (as I'm essentially doing) an acceptable strategy? I'm assuming a thread is equivalent to a virtual processor in my case, but I'm not entirely sure
- would increasing NumThreads be expected to lead to further improvements? Since I have the luxury of configuring my hardware, I could conceivably go for e.g. 64 processors and set NumThreads=20
- bonus points for other optimization suggestions
回答 (1 件)
Raymond Norris 2022 年 1 月 31 日
When you run the code serially, do you see high core activity? If not, you might not see any appreciable difference increasing NumThreads. Depends on the work the parfor loops is running. Might be worth testing it by throttling NumThreads and seeing what performance improvements you get.
Have you consider at some point the need to scale your code to a cluster/cloud?