Request exclusive use of slurm batch nodes for a parpool that is less than the total number of cores.

Hi, is there a way to tell a slurm matlab job to run in exclusive mode on 2 nodes even though the parpool is using fewer than the total number of cores? Our slurm batch nodes have 128 cores, but I am only requesting a parpool of 240, but I would still like the job to run exclusively on the 2 nodes. We are using matlab parallel server and my batch job looks like:

#!/bin/sh

#SBATCH --partition=fermi

#SBATCH --job-name=test240

#SBATCH --output=output240-%j.txt

#SBATCH --error=error240-%j.txt

#SBATCH --ntasks=1

#SBATCH --cpus-per-task=4

#SBATCH --mem-per-cpu=4g

#SBATCH --time=30:00

source /usr/share/Modules/init/sh

module load matlab

matlab -batch "addpath ('/sdf/home/r/renata/matlab/jaehyun');test240"

And test240.m has these settings:

c=parcluster;

c.AdditionalProperties.QueueName='fermi';

c.AdditionalProperties.EnableDebug=true

c.AdditionalProperties.NumNodes=2;

c.AdditionalProperties.ProcsPerNode = 121;

p=c.parpool(240);

Thanks for any insight,

Renata

Answer 1

[Raymond Norris]

Renat your MATLAB code looks similar to installations we've supported in the past. If that's true, run

c.AdditionalProperties

Do you see the RequireExclusiveNode field? If so, you can set it as such

c.AdditionalProperties.RequireExclusiveNode = true;

If the field is not there, contact me and I can help you resolve this.

[Renat Dart]

Hi Raymond, thanks for your response, can the user just include

c.AdditionalProperties.RequireExclusiveNode = true;

in the test240.m matlab file?

Renata

[Raymond Norris]

That won't be enough. That only sets the value. getCommonSubmitArgs will use it (which may or may not be implemented). If RequireExclusiveNode is not a property, it's safe to assume it's not getting used.

A shortcut is to call

c.AdditionalProperties.AdditionalSubmitArgs = '--exclusive';

Though long term, it's preferable to use RequireExclusiveNode.

[Renat Dart]

Hi Raymond, does

c.AdditionalProperties.RequireExclusiveNode = true;

then force exclusive on all slurm jobs running matlab?

Renata

[Renat Dart]

Hi again, also for the shortcut approach, would that line just be added to the existing like this:

c.AdditionalProperties.QueueName='fermi';

c.AdditionalProperties.EnableDebug=true

c.AdditionalProperties.NumNodes=2;

c.AdditionalProperties.ProcsPerNode = 121;

c.AdditionalProperties.RequireExclusiveNode = true; <-------

p=c.parpool(240);

or should the ProcsPerNode be removed?

Thanks,

Renata

[Raymond Norris]

To be clear, the "shortcut" is assigning AdditionalSubmitArgs, as such

c.AdditionalProperties.AdditionalSubmitArgs = '--exclusive';

I'm assuming that you're not supporting RequireExclusiveNode

c.AdditionalProperties.QueueName='fermi';
c.AdditionalProperties.EnableDebug=true
c.AdditionalProperties.NumNodes=2;
c.AdditionalProperties.ProcsPerNode = 120;
c.AdditionalProperties.AdditionalSubmitArgs = '--exclusive';
p=c.parpool(240);

And don't you want ProcsPerNode to be 120?

[Renat Dart]

Hi Raymond, I checked c.AdditionalProperties and there is no RequireExclusiveNode field. What do I need to do to allow that to be used if a user chooses?

Thanks,

Renata

[Raymond Norris]

Hi Renat. Contact me directly and I'll work to get this resolved for your installation.