Large training set in Semantic Segmentation runs out of memory in trainNetwork

Dear Community!

Since the following question has not been answered yet, i would give an update with more details.

https://www.mathworks.com/matlabcentral/answers/413264-large-training-set-in-semantic-segmentation

I would like to train a dataset contains 600 piece of 1208x1920 images and 50 classes.

I used the following code just changed the classes and the paths:

https://www.mathworks.com/help/vision/examples/semantic-segmentation-using-deep-learning.html

However the training runs on the following error:

Where the 1208x1920 is the image size, 50 is a number of classes and 200 is the number of the validation pictures.

With 1 piece of validation picture the training is starting

Memory: 64 GB

GPU : Titan X Pascal 12GB

We would like to know what is the best way to overcome this problem.

Answer 1

[Raunak Gupta]

Hi,

As mentioned in the example that is referenced you may need to resize the image to a smaller size that can fit into the GPU memory or you may try reducing the MiniBatchSize to a smaller value like 4 or 2. If even 1 image doesn’t fit into the memory you need to resize the image of choose a smaller network to work with or increase the GPU memory on the system. Here since the imageDatastore is used, all the validation images won’t be read from the memory instead only ‘MiniBatchSize’ number of images will be read.

Here since the image size is almost 3.4 times the size used in example, so I recommend first changing the MiniBatchSize to 2 as compared to 8 in the example.

For increasing the MiniBatchSize the best way is to increase the GPU Memory present on the system.

[Lorant Szabo]

Hello!

As you can see this bug is not releated to the GPU memory since without validational dataset the training is starts with MiniBatchSize=10.

This bug is in connection with the validational dataset size.

It occured because in the trainNetwork function tries to allocate array for the results of the validational set.

[Raunak Gupta]

Hi,

In the example that is referred the data is divided using a function partitionCamVidData that creates seperate datastore for training, testing and validation according to the split that is mentioned. As long as long the data is feed into the network using datastores the images are not loaded into the memory. Also the result array will not be of much size because it contains only accuracies and losses for each iteration (Even if the number of iteration is 10000 the overall size will 10000 x 5 as it store only 5 parameters). So, unless the validation data is loaded specifically into the code, the datasores will not load data into the GPU Memory. The data will loaded according to the batch size. Also you may explictly set the 'ExecutionEnvironment' to 'gpu'.

[Brian Derstine]

What do you mean by this: "unless the validation data is loaded specifically into the code,"?

Is there a particular code pattern that will cause the entire validation set to be loaded into memory?

Update: this is a bug in how validation datasets are handled. See: https://www.mathworks.com/matlabcentral/answers/570004-out-of-memory-error-when-using-validation-while-training-a-dagnetwork

reduce your validation dataset size and it should work.

[Brian Derstine]

also, an update in 2021b may fix this bug: "You are correct. In MATLAB R2021a, there is a bug in the Neural Network Toolbox where, depending on the workflow, if the validation data is large, you may run out of memory on the GPU. This has been reported in image segmentation and LSTM workflows.

The workaround is to reduce the validation data set size or train without validation data. Reducing the "miniBatchSize" does not fix this issue.

A patch for this bug was made in MATLAB R2021b. You may want to consider using this version of MATLAB to avoid encountering this issue." (response from matlab support)