Parallel Computing Toolbox: Underperformance AND Cannot cancel or destroy a job that was not created by this Local cluster.

I am running Win Server 2012 R2 with 2014A with PCT. I have two questions.

Question 1:

I keep getting the below warning,

    Warning: Cannot cancel or destroy a job that was not created by this Local cluster. 
    > In Local.cancelOrDestroyJob at 34
      In Local.Local>Local.hDestroyJob at 237
      In CJSJobMethods>CJSJobMethods.destroyOneJob at 70
      In CJSCommunicatingJob>CJSCommunicatingJob.destroyOneJob at 94
      In Job.Job>Job.delete at 1037
      In my_mFile at 39 
    Starting parallel pool (parpool) using the 'local' profile ... connected to 32 workers.

My code looks like,

        myCluster = parcluster('local');
        delete(myCluster.Jobs);
        delete(gcp);
        parpool('local',32); 
        A = length(myFiles);
        parfor a = 1: A
            %do stuff
        end

The reason I included "delete(myCluster.Jobs);" is that without it I get another warning, as below. I should say I only have one MATLAB IDE running at anyone time on my machine.

Warning: Found 1 pre-existing communicating job(s) created by pool that are running. You can use 'delete(myCluster.Jobs)' to remove
all jobs created with profile local. To create 'myCluster' use 'myCluster = parcluster('local')'.

What code do I need to change/include to make sure I do not get any warning?

Question 2:

I have 32 vCPU (vCPU = half a physical core, ie half of a shared core, or one hyperthread), (Intel Xeon E5-2680 v2) on my machine. The task I am trying to run is very demanding (each task takes c. 3mins to run). Hence I want to make sure the machine is maxed out.

In the MATLAB IDE I go to HOME, parallel, manage cluster profiles, edit and change "Number of Workers to start on your local machine" to 32.

I start the job. Task manager shows the total CPU at 40-50%. Going into resource monitor shows the individual matlab.exe taking up 0.2% CPU on average.

Is this right? Normally when I use PCT (on a normal/low spec machine) and don't set myCluster, parpool(local,32) etc) I get CPU useage of 100%.

When I remove the two lines:

   myCluster = parcluster('local');
   delete(myCluster.Jobs);

and then run the job, I see CPU now averages at >90%. No idea why...

thanks

[Darwin Brochero]

I think that warning appears due to PARFOR loops unsuccessfully closed in other passed sessions. But you can rename (or deleted ?) configuration file that saves history of pass parfor evaluations (in my GNU/LINUX> .matlab/local_cluster_jobs/R2014a/matlab_metadata.mat)

[Darwin Brochero]

oups!

Renaming file gives an error in new parpool sessions, so in my case I save original configuration file with some prefix (orig-matlab_metadata.mat), and when I have this warning, I copy this file as "matlab_metadata.mat" to avoid this problem.

...That's still as a not technical fix, I know, but it's works.

Answer 1

[Sean de Wolski]

The only time I have seen that is also when something failed in a previous MATLAB session (hang/out of memory/etc.)

You should be able to clear them out in the job monitor.

http://www.mathworks.com/help/releases/R2014b/distcomp/job-monitor.html