Batch jobs of similar data size and function files are taking different times for job completion, though all are started at same time.

I have a large (n x n) data, which is an input for several dependent function files.

Each element should undergo the same process, and the communication between the elements is not necessary.

So I partitioned my data into 'z' number of zones corresponding to number of cores in local machine.

Three columns and two rows.

r_zones = 2; % row zones
c_zones = 3; % column zones

rz = round(linspace(1,sx,r_zones+1)); %rz = row zones
cz = round(linspace(1,sy,c_zones+1)); %cz = column zones

Six jobs are created for six zones with batch command.

pc = parcluster();
for i = 1:r_zones
    for j = 1:c_zones
        Xa((rz(i):rz(i+1)-1),((cz(j):cz(j+1)-1))) = 1;
        job(jm(i,j)) = batch(pc,@com_sigma,3,{inputs});
    end
end

for i = 1:r_zones
    for j = 1:c_zones
        wait(job(jm(i,j)));
    end
end 

All the jobs have same size of data ( a difference of one row or column may present) and started at the same time.

But, the finishing time of each job is different and the initial job is taking longer time. And, the CPU utilisation is not more than 70%.

Answer 1

[Raymond Norris]

Not sure why the 1st task isn't finishing, but here's another approach. Keep what you have, but rather than using batch, use parfeval. Change

pc = parcluster();
for i = 1:r_zones
    for j = 1:c_zones
        Xa((rz(i):rz(i+1)-1),((cz(j):cz(j+1)-1))) = 1;
        job(jm(i,j)) = batch(pc,@com_sigma,3,{inputs});
    end
end

to

pc = parcluster();
pool = gcp('nocreate');
if isempty(pool)
    pool = pc.parpool(maxNumCompThreads);
end
for i = 1:r_zones
    for j = 1:c_zones
        Xa((rz(i):rz(i+1)-1),((cz(j):cz(j+1)-1))) = 1;
        job(jm(i,j)) = parfeval(pool,@svd,3,inputs);
    end
end

You're start one job (a parallel pool) and then sending individual tasks to it (with parfeval) rather than creating r_zones * c_zones jobs (with batch).