Remote workers still access data files from the client despite the files have been copied to the remote host beforehand

1 回表示 (過去 30 日間)
Hi,
I have two PCs, and I use Parallel Server and Parallel Computing to utilize the cores in both PCs simultaneously. Since the script I'm running load relatively large amount of data files (.mat), I copied the files to the remote host beforehand, and then use the following command:
MJSName = 'AdibMJS';
ScriptName = 'SatelliteDataProcessing';
PathInRemoteHost = 'D:\SatelliteData';
NumWorkers = 8; % 4 from each PC
C = parcluster(MJSName);
Job = batch(C, ScriptName, 'Pool', NumWorkers - 1, 'AdditionalPaths', PathInTheRemoteHost);
parfor i = 1:N
load('SomeData.mat')
% Some codes
end
By doing this, I thought I could avoid the files to be transferred during runtime through WLAN, which is slow in my setup. When I started the job, I could see that, through Task Manager of both PCs, there are Network usage by MATLAB, indicating that there are still some data being transferred through WLAN. This cause the parfor-loop to run slower compared to using less workers (e.g., 4) on just the local cluster. What I want to happen instead is for both PCs to load the data files locally and run the parfor-loop simultaneously. Note that in both PCs, the path that stores the data files is identical.
Please advise where did I do wrong? Thank you!

採用された回答

Jason Ross
Jason Ross 2021 年 2 月 10 日
It sounds like there might be some dependency analysis still going on that's sending the files over. You can turn that off by setting 'AutoAddClientPath' and 'AutoAttachFiles' to 'false' in your batch command.
See the batch documentation for more info.

その他の回答 (0 件)

カテゴリ

Help Center および File ExchangeMATLAB Parallel Server についてさらに検索

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by