MATLAB Parallel server: save output data of each worker by program independent job

The program independent job can work on 10 workers like the following example.

It is assumed that all workers are diffrent nodes each other (node01~node10).

The following code is run on the worker of node01.

% % example.m

c = parcluster;

job1 = createJob(c);

num_worker = 10;

for cnt = 1:num_worker

input_var{cnt} = cnt;

end

createTask(job1,@test_func,1,input_var);

submit(job1);

wait(job1);

result = fetchOutputs(job1);

function output_var = test_func(input_var)

A = rand(100,100) + 1i*rand(100,100);

output_var = A*input_var;

I want to save the results from all workers.

If the system memory of computing the results is over the limit, we can't acquire and save these results.

So, I try to save each result inside the function, 'test_func.m', as follows.

function [] = test_func(input_var)

A = rand(100,100) + 1i*rand(100,100);

output_var = A*input_var;

save(sprintf('filename_%d.mat',input_var),'output_var');

If the function is changed as above, the result of each worker is saved in the following path,

'C:\ProgramData\MJS\Checkpoint\node(number)._worker01_mlworker_log\matlab\work'.

I want to save these files on the other path, not a default path,

like the path of node01, '\\node01\E:\folder_name\'.

The follows have error, not to find the path on the other worker except to headnode (node01).

function [] = test_func(input_var)

A = rand(100,100) + 1i*rand(100,100);

output_var = A*input_var;

save(sprintf('\\node01\E:\folder_name\filename_%d.mat',input_var),'output_var');

I wonder how to save output data of each worker on user-defined path of headnode.

Please, help me.

Thank you.

Answer 1

[Raymond Norris]

Try the following

function [] = test_func(input_var)

% Store the local hostname for future runs
persistent HN
if isempty(HN)
    [FAILED, HN] = system('hostname');
    if FAILED
        HN = getenv('COMPUTERNAME');
    end
    HN = strtrim(HN);
    if isempty(HN)
        error('Failed to get hostname')
    end
end

A = rand(100,100) + 1i*rand(100,100);

output_var = A*input_var;

% Need to escape each of the backslashes.
% E:\ won't work.  Use E$
save(sprintf('\\\\%s\\E$\\folder_name\\filename_%d.mat',HN,input_var),'output_var');

[Lee Jong Hyun]

I try it.

Thank you.

[Lee Jong Hyun]

I tried it, but I get an error.

As the path defined by 'sprintf' is not existed, it cannot create the file.

And I have a question.

As shown in your answer, 'HN' is hostname of each worker.

However, I want to save data from each worker on the disk of node01.

For example,

If the data from node02 is computed, it should be that the path to save the data is the path of node01.

Can I save the data on the path of node01?

Thank you.

[Raymond Norris]

You're saying you want (notice the E$ and not E:)

save(sprintf('\\\\node01\\E$\\folder_name\\filename_%d.mat',input_var),'output_var')

Is node01 reachable by each machine?

[Lee Jong Hyun]

The shared drive (E) using a local network of node01 can be accessed from each node.

As I access the directory in the drive on MATLAB of node02, the script is as follows,

save('Z:\folder_name\filename.mat','variable');

which 'Z' is the shared drive of node01.

This example is not using program independent job, just directly run on node02.

I coded the same path for using the function 'save', when I used the independent job at the first time. This way had error. So, I hsa tried other approaches such as changing the path name.

'\\(ip address of node01)\E:\(folder_name)\(filename.mat)'

or

'\\(hostname of node01)\E:\(folder_name)\(filename.mat)'

[Raymond Norris]

Just so that I understand, you have 10 workers running on 10 nodes. Each worker is to write a result file to the same machine (node01). This machine can be accessed via the E:\ drive. When you try the following

save(sprintf('\\node01\E:\folder_name\filename_%d.mat',input_var),'output_var');

MATLAB throws an error. What error does MATLAB throw? Does each worker throw the same error?

I don't believe this will work for several reasons.

1. For starters, is "E" a mapped drive or a physical drive? If it's a physical drive, then \\node01\E: isn't proper syntax. I believe this should be \\node01\E$.
2. If E is mapped drive, then you ought not need to specify \\node01 (since that's already apart of "E")
3. But I don't think E:\folder_name\... will work because drive mappings are specific to a user, not a machine. Hence, you need to go back to the UNC path (\\node01\E$)

What happens when you try

save(sprintf('\\node01\E$\folder_name\filename_%d.mat',input_var),'output_var');

[Lee Jong Hyun]

Thank you very much!