Is it possible to prevent parpool('Processes') to create a large number of open files?

I have noticed the following behavior of the parpool-function in Matlab (2023a). When I use something like

pool = parpool('Processes', 128)

it return with an error

Starting parallel pool (parpool) using the 'Processes' profile ...

Parallel pool using the 'Processes' profile is shutting down.

Error using parpool

Parallel pool failed to start with the following error. For more detailed information, validate the profile 'Processes'

in the Cluster Profile Manager.

Caused by:

Error using parallel.internal.pool.AbstractInteractiveClient>iThrowWithCause

Failed to initialize the interactive session.

Error using parallel.internal.pool.AbstractInteractiveClient>iThrowIfBadParallelJobStatus

The interactive communicating job failed with no message.

This error is due to the fact, that on the used Linux server, the number of files is limited

$ ulimit -n

131072

and it seems that Matlab is opening >1000 files per worker requested for the pool. This seems to be an unreasonable large number of files imho so my question is if I can somehow prevent this?

I know that I can of course increase the limit on the number of files in the OS. I also noticed that

pool = parpool('Threads', 128)

does not suffer from this problem (and the pool has also a much short startup time), but in parfor-loop a piece of Mex-compiled code is used and this is not possible for a pool of threads:

Use of MEX functions is not supported on a thread-based worker.

So an alternative question would be how to use a MEX function on a thread-based worker?

Answer 1

[Paras Gupta]

Hi Stefan,

I understand that you get an error when you start a parallel pool using 'Processes' with 128 workers.

This seems to be a known bug in MATLAB R2023a for cases when the number of workers is more than 100. Please refer to the following bug report for a workaround to resolve the issue:

https://www.mathworks.com/support/bugreports/details/2968710.html

Also, using a MEX function on a thread based worker seems to be a limitation of MATLAB as of the latest release R2024a. You may contact MathWorks Technical Support for more information on the same.

Hope this answers your query.