Can I run multiple instances (say, 2) of matlab independently on the same computer, all using parfor and (ideally) different cores?

I have many cores (say 20), but some scripts only require a fraction (say 6). Does it make sense to run multiple scripts in different matlab instances? What do I need to look out for so they do not bog each other down?

Answer 1

[Walter Roberson]

Yes, this is something that can be reasonable to do.

What you would want to watch out for is not using more total cores than physical cores you have (with hyperthreading turned off), minus one for your OS to do all its background tasks like checking email. (And if you are using Windows, minus another one to pay the Microsoft Tax.)

Splitting a system with a number of cores to do different independent tasks is the basis of running "supercomputer" servers with Distributed Computing; https://www.mathworks.com/products/matlab-parallel-server.html . But since it is all local to you, you might at most want to create multiple parallel profiles.

[Alexander]

I have a very primitive understanding about the cluster setup. I will look into that. Thanks!

[Florian]

Thank you Walter!

If I might tag onto this question: I have a local computer with 28 cores that I want to use simultaneously with a colleague. We both have individual academic licences.

If we both run parallel pools in separate windows user accounts (and separate Matlab instances) I run into problems where workers cannot locate files to write into using fprintf.

However, if I run two parallel pools in separate Matlab instances in the same windows user account, this problem does not occur.

Do you have any insights into why this might be happening? To put it amateurishly, does Matlab "not see" parallel pools created by other user accounts?

Answer 2

[Matt J]

Does it make sense to run multiple scripts in different matlab instances?

No, that would be a reinvention of what the Parallel Computing Toolbox already does for you, but requiring more manual work on your part. Just run all the scripts in the same Matlab instance, but in parallel, e.g., using parFeval. Let the toolbox worry about scheduling the cores.

[Alexander]

But they are different problems. Think optimizations where the derivative is done in parallel, but there are only 6-8 parameters. Obviously, I cannot run them in parallel. But it also seems a pity having idle resources.

[Alexander]

The thing is they are really completely independent tasks that require some oversight and (possibly) interventions. So, I calibrate the same model to different parameters, and since I do not have a good idea about starting values I first calibrate some moments to get into the ballpark. Then I tell the optimizer to stop, and include more moments that I would like to match. etc. I do not want to stop one optimization just because another one is good enough to go to the next stage.