How to submit an array job in slurm that calls a matlab compiled application that contains mpirun?

slurm - array job - mpiexec - mcr - mpirun

The objective is to run in parallel a program that contains mpirun executions itself, on a cluster with slurm (with a job array).

The program compiled with matlab looks something like this (no hyperthreading allowed with openfoam):

option 1:

%% myprogram 
function out = myprog(input)

    prepareOpenFoamCase;
    
    system('mpirun -np 16 openfoam_application -parallel')
    
    out = readOpenFoamOutput;
    writetabel(out,'output.txt');
    
end

The first slurm script that workw on the cluster looks like this (modules are omitted for clarity):

%% slurm script
% #SBATCH --nodes=1 
% #SBATCH --ntasks=1 
% #SBATCH --array=0 
% #SBATCH —cpus-per-taks=16 

%myprogram calls mpirun -np 16 
./myprog 

option 2:

The second program is the same as above except for the execution of the openfoam application, which does not call mpirun.

%% myprogram 
function out = myprog2(input)

    prepareOpenFoamCase;
    
    system('openfoam_application')
    
    out = readOpenFoamOutput;
    writetabel(out,'output.txt');
    
end

and the script looks like this:

%% slurm script for myprogram2
% #SBATCH --nodes=1 
% #SBATCH --ntasks=48 
% #SBATCH --array=0 

%myprogram2: similar to myprogram but no mpirun inside 
mpiexec -n $SLURM_NTASKS ./myprog2

objective:

The objective is to have something like this:

%% slurm script goal 
% #SBATCH --nodes=1 
% #SBATCH --ntasks=3 
% #SBATCH --array=0-9
% #SBATCH —cpus-per-taks=16 
% #myprogram calls mpirun -np 16  
mpiexec -n $SLURM_NTASKS ./myprog 

However, I realize this is not correct with a mpiexec that wraps a mpirun, and obviously when the execution reaches the mpirun -np in myprog it stops.

Does this happen because this nested approach is not supported or is it maybe just formulated badly?

Trying to execute in parallel the application that contains mpirun like this :

%% slurm script goal 
% #SBATCH --nodes=1 
% #SBATCH --ntasks=1 
% #SBATCH --array=0-9
% #SBATCH —cpus-per-taks=16 
% #myprog calls mpirun -np 16  
./myprog 

resulted in But this would mean that I allocate only 16 cores per node, or would it still be distributed effectively by slurm?

Could approaches with hostfiles and application context help, but I am not sure if the objective is feasibe at all and if these are the right approaches, or if there are other effective options?

The following is an approach with localhost and hydra that should allow me to run myprog three times on the same node. But still, is it possible to formulate for an array job?

%% slurm with hostfile and hydra 
% First create a hostfile with the entry localhost

for I in `seq 0 95` 
do 
    echo localhost
done >localhost

%#prepare the MPI environment
unset I_MPI_HYDRA_BOOTSTRAP
unset I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS
export I_MPI_FABRICS=shm
export I_MPI_HYDRA_HOST_FILE=localhost

%#now you can start multiple mpiexecs with the node
%#load a test program
module load placementtest
I_MPI_PIN_PROCESSOR_LIST=0-15   ./myprog >OUT.1 &
I_MPI_PIN_PROCESSOR_LIST=16-31  ./myprog >OUT.2 &
I_MPI_PIN_PROCESSOR_LIST=32-47  ./myprog >OUT.3 &
wait

Thanks in advance!

Sebastiano