parallel computing toolbox - how to directly talk to the Workers?

I see for example that the Batch/Workers concept allows to sumbit the jobs to the scheduler which, in turn, schedules the jobs to the available workers(cores). Is there a way that user can skip the scheduler and talk to the workers directly? In other words I am looking how to simply statically map the jobs to the cores, to have full control what is going on rather that letting Matlab scheduler to do that under the hood.

Answer 1

[Raymond Norris]

You can use different schedulers; however, you need to call the parpool command to start workers. In order to start the workers, they need to be scheduled to run. Hence the need for MATLAB to interface with a scheduler.

[Dusko Vujadinovic]

Thanks a lot for the answer. It is good to hear that is possible to replace the scheduler with own one. However it opens a lot of new questions how to replace existing one. So any links for that would be highly appreciated.

[Raymond Norris]

If you have your own scheduler (e.g. PBS), then this link should help

https://www.mathworks.com/help/matlab-parallel-server/integrate-matlab-with-third-party-schedulers.html

However, if you don't have a scheduler already, then I would suggest keeping with the MATLAB job scheduler rather than managing a scheduler solely for the purpose of MATLAB.

[Dusko Vujadinovic]

I have tried to ask the right question by being short but apparently to get the answer I am looking forward, I need to explain the context.

My idea is to try using Parallel Computing Toolbox to run a wireless modem (relatively complex) application on a multicore (Linux) machine. Thus, the modem application is a real time signal processing application that contains a number of signal processing jobs that need to be scheduled on the cores. Some of those jobs can be run in parallel, some of them need to wait for multiple other jobs to provide results before themselves being started etc, some of them can be run serially independently from other jobs. So, the point is that the scheduling of those jobs is completely application specific and it is crucial to have a full control over entire process to meet the real time requirements, to design the modem to use the resources in maximum optimum way (very complex composite criteria). From the other side the application is deterministic is terms of the jobs that need to be done, so instead of dynamic scheduling, the static mapping of the jobs to cores (workers) should be enough. Everything is supposed to happen on a single (multicore Linux) machine.

Ideally, I do not want to reinstall Matlab to put my own scheduler, since only for the modem application I need to have full control over the jobs to do the cores mapping. So, when the modem is not run Matlab can use own scheduling. When the modem application runs it exclusively own all resources, so other Matlab applications should be blocked to run - until the modem application terminates its work.

So how realistic is that I can use Matlab on this way - it might be that the replacing scheduler is not the way forward, but there is some other way?

It could be possible that the modem application can be run with the existing Matlab multicore scheduler, but I need to have full visibility about how the scheduler runs the jobs - I need to see full history which job was run on which core and during which time period. is that available?