Question about Resource Allocation in MATLAB

I use confidential data available through a secure server and have been repeatedly encountering the following issue related to the early termination of a Matlab program/M file:

"PBS: job killed: cpus 34.1 exceeded limit 30 (burst)"

If I were to increase the number of CPUs to 40 or even 50, the M file would run for a while (20 minutes or so), and the job would then be killed off with the same message displayed.

The issue is not caused by errors in the M file itself, as the same program runs to completion in about 3-4 hours on my own Mac laptop with only 2 CPUs (when the program is tested on fake/simulated data). The issue (as I perceive it) seems to be caused by insufficient/incorrect resource allocation. However, this justification seems rather surprising as I previously thought that MATLAB is supposed to ``know" the number of available cores. I would really appreciate any suggestions/thoughts on how to get the M file running to completion.

As an addition clarification, I have run into issues with MATLAB though the server before, but in that case, I just requested 3 or 4 CPUs (as opposed to just 1 CPU). Maybe this particular case is different because the program (which searches for all the local maximums of a specific function) is more "computationally intensive"? It certainly has multiple nested for loops.

[Walter Roberson]

MATLAB does not know the number of "available" cores: it only knows the number of physical cores, together with information as to whether hyperthreading is enabled.

parpool() with the default configuration uses a pool size equal to the number of physical cores.

For a given pool size, pool creation creates that many processes.

Management of the processes, as to which is active, and even of processor affinity, is up to the operating system.

[Sherry Wu]

Thanks Walter. I can think of only one solution put an upper bound on the number of threads using maxnumCompThreads. Are there any alternatives?

[Walter Roberson]

Ah, I realize that I assumed you are using parpool, which is not necessarily the case.

If you have sufficiently large vectorized mathematics, then MATLAB will delegate the work to the third-party LAPACK library which will create one thread for each hardware core, except that it is also limited by maxnumCompThreads. So using maxnumCompThreads might be the way to go.

[Sherry Wu]

Hi Walter,

I have tried adding in resource constraint by specifying "LASTN=maxNumCompThreads(..)" right before running the ``computationally intensive" program SearchAllLocalMaximum.m. However, I seem to get the same error after a while. Any suggestions?

More specifically, here is the sequnce of commands:

-When first logging onto the server, type

"qsub -I -X -l select=1:ncpus=40:mem=40gb"

-Open Matlab and run Script1.m. Within Script1.m, type the line "LASTN=maxNumCompThreads(40)"

right before "SearchAllLocalMaximum.m"

__________________________________________

After running Script1.m interactively for ~10 minutes, I get the error message

"PBS: job killed: ncpus 46.6 exceeded limit 40 (burst)"

[Walter Roberson]

It might be worth asking for max comp threads of say 32 with qsub cpu count of 40. That will give a hint of whether the thread control is having any effect.

I am not clear on whether you are using parpool?

Answer 1

[Sherry Wu]

Hi,

I asked max comp threads for 10 with qsub cpu count of 40 yesterday. The good news is that the programs did not crash after running them interactively for more than one hour, whereas they used to get killed after 15 minutes.

What I find somewhat confusing is that when I checked the number of threads after an hour or so ("N = maxNumCompThreads returns the current maximum number of computational threads N."), the returned value was 25 when I specified 10 beforehand.

I am not familiar with the parpool feature (Is it supposed to speed out for loops?), and so I guess I wasn't using parpool. Relatedly, I was surprised that Matlab would use multiple threads for a single M file that I did not explicitly request running parallel jobs. My guess is that some commands in the M file are implicitly parallelized in Matlab?

Thanks!

[Walter Roberson]

(for some patterns of computation) "If you have sufficiently large vectorized mathematics, then MATLAB will delegate the work to the third-party LAPACK library which will create one thread for each hardware core, except that it is also limited by maxnumCompThreads."

parpool together with parfor or spmd or batch or parfeval are part of the Parallel Computing Toolbox, and provide for explicit parallelization. It is common that explicit parallelization turns out worse than implicit parallelization (by way of LAPACK), but there are some kinds of algorithms for which explicit parallelization works well.