Running trainNetwork() starts parallel pool in Matlab 2023b even when options.Ex​ecutionEnv​ironment = 'cpu'

It seems the behaviour of trainNetwork has changed. I have a for loop that performs a grid search of models' hyper-parameters. After updating to version 2023b, parallel pool is being started whenever trainNetwork is invoked. Is this a bug or trainNetwork behaviour changed?

[Artem Lensky]

Unchecking "Automatically create a parallel pool.." does not start the pool automatically anymore. This behaviour seems to be different from previous versions.

[Artem Lensky]

Now if I run the following code

delete(gcp('nocreate'));

parpool(numTrials);

accuracies = zeros(1,numTrials);

parfor l = 1:numTrials

[net, ~] = trainNetwork(trainSetX, trainSetY, layers, options);

YPredicted = classify(net, testSetX)';

accuracies = accuracies + circshift(eye(1, numTrials), l - 1) .* cellfun(@(y1, y2) sum(y1 == y2)/length(y1), YPredicted, testSetY);

numberOfWeights{l} = computeNumberOfWeights(net);

end

I am getting this error

Error using trainNetwork
Automatic creation of a parallel pool is disabled. To run in parallel, open a parallel pool or enable automatic creation in the Parallel Preferences menu.

any help is much appreciated. I followed this example https://au.mathworks.com/help/deeplearning/ug/use-parfor-to-train-multiple-deep-learning-networks.html

[Artem Lensky]

Even if I specify the pool to run on, it still generates the same error. So this what I do:

po = parpool(numTrials);

f(1:numTrials) = parallel.FevalFuture;

for l = 1:numTrials

f(l) = parfeval(po, @trainAndTestNetworks, 3, trainSetX, trainSetY, layers, options);

end

Where the trainAndTestNetworks is defined as follows

function [net, accuracy, numberOfWeights] = trainAndTestNetworks(trainSetX, trainSetY,layers, options)

[net, ~] = trainNetwork(trainSetX, trainSetY, layers, options);

YPredicted = classify(net, testSetX)';

accuracy = cellfun(@(y1, y2) sum(y1 == y2)/length(y1), YPredicted, testSetY);

numberOfWeights = computeNumberOfWeights(net);

end

and here is the erorr

State: finished (unread)

Error: Automatic creation of a parallel pool is disabled. To run in parallel, open a parallel pool or enable automatic creation in the Parallel Preferences menu.

trainNetwork (line 191)

gridSearch>trainAndTestNetworks (line 78)

Looks like a bug.

[Artem Lensky]

Settign the aforementioned checkbox and running the code generate the following error:

1x3 FevalFuture array:

ID              State        FinishDateTime               Function  Error
---------------------------------------------------------------------------
1   151  finished (unread)  22-Sep-2023 21:17:21  @trainAndTestNetworks  Error
2   152  finished (unread)  22-Sep-2023 21:17:21  @trainAndTestNetworks  Error
3   153  finished (unread)  22-Sep-2023 21:17:20  @trainAndTestNetworks  Error

and the error

Error: A parallel pool cannot be started from a worker, only from a client MATLAB.
trainNetwork (line 191)

[Artem Lensky]

In function

function executionSettings = setupExecutionEnvironment(options)

there is this if statement

% Validate PCT if needed. It is not needed only when the execution
% environment is set as CPU and dispatch in background is false
if isPoolNeeded || options.ExecutionEnvironment == "gpu"
    nnet.internal.cnn.util.validatePCTIsInstalled(options.ExecutionEnvironment);
end

for some reason even when options.ExecutionEnvironment set to "cpu", isPoolNeeded is set to true, which leads to executing the pool even when there is no need for it.

[Artem Lensky]

I think I identified the problem, option DispatchInBackground was set to True expcplicitly. Surprisingly, it didn't execute the pool in previous Matlab version.

Answer 1

[Artem Lensky]

Whenever option DispatchInBackground set true, trainNetwork executes parallel pool.

If I am not mistaken this behaviour was different in 2023a, I already deleted it so can't verify, but it would be nice if someone at Mathworks would check wherther

function executionSettings = setupExecutionEnvironment( opts, X, precision, networkInfo )

contains these lines

% In DLT if the training datastore is set up for background prefetch,
% enable background prefetch
backgroundPrefetch = iUseBackgroundPrefetch(X) || opts.DispatchInBackground;

[Joss Knight]

This looks like a bug in R2023a and before for DispatchInBackground. Before 23b it wasn't checking the pool auto-create option before opening a pool.

If you want to use DispatchInBackground inside parfor then you should open a thread pool manually:

parpool('threads');

This should support your data dispatch because you have in-memory data. However, I doubt there's any performance benefit from this.

Also, to run multiple parallel training trials on a cluster, use Experiment Manager.

[Artem Lensky]

Thanks for the prompt reply and clarification Joss. Very much appreciate it.

I've used Bayesian Optimisation extensively, unfortunately it was less interpretable than a grid search. The experiment manager does not have (or maybe I’ve missed) a simple grid search.

For example, with my current implementation I can simply specify what grid points to go over e.g.

hParameters = struct(...
    'numLabels',        3,...
    'numChannels',      numChannels,... 
    'dropoutFactor',    0.125,...
    'filterSize',       [5, 11, 15],...
    'numFilters',       2.^(3:5),...
    'typeOfUnit',       {"lstmLayer", "gruLayer", "bigruResults", "bilstmResults"},...
    'numBlocks',        1:4);

numTrials = 3;

results = gridSearch(@constructRNN, XSearchTrain, YSearchTrain,...
                                    XSearchTest, YSearchTest,...
                                    hParameters, options, numTrials,...
                                    ["numBlocks", "numFilters", "typeOfUnit"]);

I would also like to run several experiments/trials for exactly the same hyperparameters, but with different random seeds, this is neccessary to test for stability/sensitivity to initial weights initialisation.

If I overlooked this functionality of the experiment manager, plesae do let me know. Thanks again!

[Joss Knight]

Experiment Manager can do an parameter sweep in whatever way you like, and the parameters can be anything you like, so rng seeds for instance.

Implementing a specific grid search algorithm (sweeping one parameter at a time) may require more manual intervention I'll admit. But you might still find it saves you time.

[Artem Lensky]

Thanks for the reply. I might give it another go. I am surprised the grid search hasn't been implemented as an out of the box feature given that it is one of the basic parameter search approaches. Perhaps in 2024 version :)