How to avoid repeatly loading parallel pool when running .exe matlab file in cmd

I am using parfor in a matlab program.

Then I generate a excutable .exe file using "mcc -m " command on that program.

Later, when we I call the .exe file in cmd, the program load parallel pool everytime:

"Starting parallel pool (parpool) using the 'Processes' profile ..."

It takes so much time since I need to repeatly call it.

Do you know any way to avoid loading parpool every time when I call it? Can I preload it in cmd? Thank you.

[Damian Pietrus]

Hello Mingkang,

The answer that Hitesh provided is a great first step to try if you're only calling the .exe once. However if you're independently calling the .exe multiple times, then a separate parallel pool will open every time you call the executable. Instead of launching the .exe more than once, it might be possible to modify your code to accept multiple inputs so you can run all of your iterations in one instance of the executable and therefore one parallel pool.

[Mingkang]

Hello Damian,

Thank you for your coment. I understand it. However, the .exe file is an important basic function which should be called in many different programs. I have to call it many times outside.

Do you know some way that I can preload parfor pool in cmd. Or I can use system() function to call it in MATLAB environment, will it help? Thank you.

[Damian Pietrus]

When a processes parpool is launched, multiple separate MATLAB processes are initiated. The client then connects to these separate processes and the pool is established. Since each instance of your .exe is a separate client, you will always end up opening a pool every time you launch the file.

If the pool startup time is your main issue, there is one potential option. If the code in your parfor is thread safe, you can launch a thread pool instead of a process pool. Here, the workers run as a thread in the main MATLAB processes and share memory, with the added benefit of quicker startup. There's some more info here:

https://www.mathworks.com/help/parallel-computing/choose-between-thread-based-and-process-based-environments.html

Note that while thread workers are supported in standalone applications created using Compiler, not all functions are supported in a thread pool.

[Mingkang]

It is a good idea. However, I check "Select Parallel Environment" in the link you provided. It said if I want the program run in different machines, I should use process pool instead of thread pool?

I hope that the .exe file can run on other machines. Can I still use the thread pool?

[Steven Lord]

I hope that the .exe file can run on other machines. Can I still use the thread pool?

To clarify, do you want to be able to run that executable on other machines (say by sending it to one of your colleagues and having them run it on their computer) and have it execute code in parallel on that machine, or do you want to be able to run that executable on a machine (could be yours, could be a colleague's) and execute code in parallel on that machine and/or other machines? Either of those could fall into "run on other machines" but the distinction may be important for your question.

[Mingkang]

Hi @Steven Lord and @Damian Pietrus, the first is right. I will send the exe file to my colleagues. And he/she will run it on his/her own machine locally.

BTW, I tried the thread pool, it run on my machine well and much faster. I re-compile the exe file, it is good on my machine now. Thanks a lot for the solution. Hope it can also run on other people's machine.

[Damian Pietrus]

As long as the code ran as expected on your local machine, you should be able to have it run separately on your colleague's machines as well. A thread pool is limited to one physical machine at a time, meaning that both you and your colleagues can separately run the code on your own systems. Where a thread pool would not be supported would be taking one instance of your application and having the pool open across multiple machines.

[Mingkang]

I see, thank you for your help.

Answer 1

[Damian Pietrus]

Moving from comments to an Answer:

When a processes parpool is launched, multiple separate MATLAB processes are initiated. The client then connects to these separate processes and the pool is established. Since each instance of your .exe is a separate client, you will always end up opening a pool every time you launch the file.

If the pool startup time is your main issue, there is one potential option. If the code in your parfor is thread safe, you can launch a thread pool instead of a process pool. Here, the workers run as a thread in the main MATLAB processes and share memory, with the added benefit of quicker startup. There's some more info here:

https://www.mathworks.com/help/parallel-computing/choose-between-thread-based-and-process-based-environments.html

Note that while thread workers are supported in standalone applications created using Compiler, not all functions are supported in a thread pool. Also note that a single instance of your application can only run on your local machine -- a threadpool cannot be opened across multiple compute nodes

Answer 2

[Hitesh]

Hi @Mingkang,

The issue might occur if the code is creating new “parpool” everytime without checking the existence of the existing “parpool”. To avoid creating new “parpool” and use the existing instance you need to check the existence of existing parpool. You can add the below condition to check for any active "parpool" before calling "parfor". Kindly refer to the below code for an example:

if isempty(gcp('nocreate')) 
    parpool('Processes'); 
end 

Please find below the description of the functions used in above code example.

• gcp('nocreate'): This command checks if a parallel pool is currently active. The "nocreate" option ensures that it only checks for an existing pool and does not create a new one if none exists.
• isempty(...): This function checks if the result of "gcp('nocreate')" is empty, meaning no parallel pool is currently running.
• parpool('Processes'): If no parallel pool is found, this line starts a new parallel pool using the 'Processes' profile.

For more information on "parpool" and "gcp('nocreate')", refer to the below MATLAB Documentation:

parpool: https://www.mathworks.com/help/parallel-computing/parpool.html

gcp('nocreate'): https://www.mathworks.com/help/parallel-computing/gcp.html

[Mingkang]

Hi @Hitesh,

Thank you for your reply. I need to call an .exe file with parfor many independently in cmd. Everytime, it load and unload the parpool once. I hope to avoid it.

Answer 3

[Walter Roberson]

The official way to reduce the initiation time is to use the MATLAB Production Server product; it keeps pools "warm" so there is little time spent initializing pools.

If you do not use MATLAB Production Server, then there is no solution to your difficulty.

[Mingkang]

I see, thank you for you reply. Then I suppose the only way is convert the program into python.