Can't find file on worker - Parallel Programming

5 ビュー (過去 30 日間)
Thomas Clifford
Thomas Clifford 2021 年 11 月 11 日
コメント済み: Raymond Norris 2021 年 11 月 13 日
Hi, I'm attempting to parallelize part of my code to speed up the process. But have become stuck when one of the attached files cannot be found by a worker. The file is being opened and modified on each driver for calculation before being reverted back to it's original state. The missing file is the "hydroNet.NetworkFile". Do I have to create a duplicate file for each individual worker?
parpool('local','AttachedFiles',{'epanet2.dll',hydroNet.NetworkFile});
spmd
folder = getAttachedFilesFolder(hydroNet.NetworkFile);
if not(libisloaded('epanet2'))
loadlibrary('epanet2');
end
oldFolder = cd(folder); % Change to that folder
[OK,output] = system(x(hydroNet,pumpCombo,nCluster,pumpCluster,idxSysLink,logger));
cd(oldFolder); % Change back to the original folder
end

回答 (1 件)

Raymond Norris
Raymond Norris 2021 年 11 月 11 日
You mean to say that folder is empty?
At the top of the spmd, add this
spmd
getAttachedFilesFolder
hydroNet.NetworkFile
getAttachedFilesFolder(hydroNet.NetworkFile)
...
end
  4 件のコメント
Raymond Norris
Raymond Norris 2021 年 11 月 13 日
You can have many processes reading the same file, but need to have semiphore or another way to have simultaneous writes (e.g. parallel database). I'm not sure why with one worker it will on occasion throw an error.
In regards to your situtation, each worker ought to be able to read from the same file, but then they might write to their own file, followed by a way to "true" up the output. Or maybe send everyone their local part and then write to one file. For example (I'm leaving off error handling)
spmd
% Import data
fid1 = fopen('input.txt','rt');
A = fscanf(fid1,'%s');
% Modify data
len = length(A);
variantB = A(randperm(len));
% Send each local part to everyone
replicatedB = gcat(variantB);
% Write to disk one output file
fid2 = fopen('output.txt','wt');
fprintf(fid2,[repmat('%c',1,len) '\n'],replicatedB);
fclose('all');
end

サインインしてコメントする。

カテゴリ

Find more on Startup and Shutdown in Help Center and File Exchange

製品


リリース

R2021b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by