adapting pattersearch algorithm to use with distributed computing server.

I use a pattern search algorithm in the simulink toolbox to estimate a set of parameters.

Let's say I have a matlab file that opens a matlabpool and calls pattern search algorithm:

- main.m {

   %main calls pattern search
   matlabpool open

   options = psoptimset('UseParallel', 'always')

   X0 = [0.5 0.5];   % Starting point
  [x,fval] = patternsearch(@simple_objective(x),X0)
}

I send it to the hpc cluster using the pbs file as follows:

% PBS file

    #PBS -l nodes=12
    #PBS -l walltime=240:00:00
    #PBS -j oe
    #PBS -l pmem=5gb
    #PBS -q starxf
    cd ..
    matlab -nosplash -nodisplay -r main.m

How can I change this file to use the distributed computing toolbox.

I guess I could not just change the last line to dmatlab ? i.e.

    cd ....
    dmatlab main.m

I understand how simple examples work when a parfor loop is transformed using createTask and then an *.m file is called with dmatlab. But I am new to parallel computing and cannot really wrap my mind about how to do it using the distributed matlab server.

Answer 1

[Edric Ellis]

Firstly, you need to set up a parallel configuration to allow Parallel Computing Toolbox to talk to your PBS cluster. You may need to refer to the MATLAB Distributed Computing Engine documentation to do this, http://www.mathworks.com/help/mdce/index.html. Once you've validated you cluster profile, you should be able to simply remove the 'matlabpool open' line from main.m, and then submit a batch job like this:

cluster = parcluster('myPBSProfile');
job = batch(cluster, 'main', 'Pool', 11);
wait(job);
load(job)

See also the documentation for batch.

[lena]

Thank you that worked!

I was also wondering if some one could explain the difference between nodes and processors in the following context.

I used to send my estimation routine (see the example with pattern search) to the hpc cluster using the pbs file example above. The time to evaluate the objective function (i.e. @simple_objective(x)) would increase by 10-15 times compared to the time taken by the same objective function at the same values of x. Folks at hpc claimed that the reason of the slowdown lies in the first line:

#PBS -l nodes=12.

This line somehow implies that i am requesting 12 nodes on the same cpu and as a result resources are spread thin across parallel loops.

They suggested using the distributed matlab server and using

#PBS -l nodes=1:ppn:12  
(which request 12 processors over 1 node)

     #PBS -l nodes=4:ppn=2
An explanation for dummies or a reference for newbies would be greatly appreciated.