large training dataset leads to error "out of memory. Type "help memory" for your options. caused by: out of memory. Type "help memory " for your options".

hello! I am currently working on semantic segmentation project following the code mentioned in link

"https://www.mathworks.com/help/vision/ug/semantic-segmentation-using-deep-learning.html".

I am using 2500 original images and 2500 labeled images with 720x960x3 size and 50 classes to train the semantic segmentation (Resnet network) on RTX 2080Ti, 11GB GPU, but I am facing the error, " out of memory. Type "help memory" for your options. caused by: out of memory. Type "help memory " for your options".

Thankyou in advance.

[Sergey Kasyanov]

First of all, try to load training data in some steps with small sets of images.

[Marya Rasib]

Sir! Can you please tell me how can i do that?

[Marya Rasib]

And please suggest some another solution as i want to train images on 2500 images at once

[Sergey Kasyanov]

Sorry, I can't find any way to load images by sets.

Answer 1

[Joss Knight]

Have you followed the following advice from the example?:

"The images in the CamVid data set are 720 by 960 in size. Image size is chosen such that a large enough batch of images can fit in memory during training on an NVIDIA™ Titan X with 12 GB of memory. You may need to resize the images to smaller sizes if your GPU does not have sufficient memory or reduce the training batch size."

In your case you are running out of CPU memory. Perhaps you are attempting to load your entire dataset into MATLAB as a single variable rather than storing the data in files?

[Marya Rasib]

here i am gona paste a whole code with the functions! please let me know where i am doing wrong

{

clear()

resnet18();

%load dataset

segmentation = fullfile(pwd,'CamVid');

imgDir = fullfile(segmentation,'images');

imds = imageDatastore(imgDir);

I = readimage(imds,10);

I = histeq(I);

imshow(I)

%Load CamVid Pixel-Labeled Images

classes = [

"sky"

"bridge"

"building"

"wall"

"colum_pole"

"traffic_cone"

"electric_pole"

"street_light"

"traffic_light"

"paved_roads"

"unpaved_roads"

"solid_yellow_lanes"

"pavement_sidewalk"

"concrete_barrier"

"fence"

"police_vehicle"

"construction_vehicle"

"military_vehcles"

"fribrigade"

"traditional_truck"

"container_truck"

"hino_bus"

"traditional_bus"

"car"

"van"

"mini_van"

"Rickshaw"

"Cart"

"pickup"

"mini_pickup"

"Tanker"

"ambulance"

"Motorbike"

"tree"

"vegetation_misc"

"sign_board"

"Speed_limit"

"Advertisement"

"Direction_board"

"Distance_board"

"Pedestrian"

"mountains"

"grassy_road_divider"

"bicycle"

];

labelIDs = camvidPixelLabelIDs();

labelDir = fullfile(segmentation,'label');

pxds = pixelLabelDatastore(labelDir,classes,labelIDs);

%Read and display one of the pixel-labeled images by overlaying it on top of an image.

C = readimage(pxds,10);

cmap = camvidColorMap;

B = labeloverlay(I,C,'ColorMap',cmap);

imshow(B)

pixelLabelColorbar(cmap,classes);

%Analyze Dataset Statistics

tbl = countEachLabel(pxds);

frequency = tbl.PixelCount/sum(tbl.PixelCount);

figure

bar(1:numel(classes),frequency)

xticks(1:numel(classes))

xticklabels(tbl.Name)

xtickangle(45)

ylabel('Frequency')

%prepare training and testing data

[imdsTrain, imdsVal, imdsTest, pxdsTrain, pxdsVal, pxdsTest] = partitionCamVidData(imds,pxds);

numTrainingImages = numel(imdsTrain.Files);

numValImages = numel(imdsVal.Files);

numTestingImages = numel(imdsTest.Files);

% create network

% Specify the network image size. This is typically the same as the traing image sizes.

imageSize = [720 960 3];

% Specify the number of classes.

numClasses = numel(classes);

% Create DeepLab v3+.

lgraph = deeplabv3plusLayers(imageSize, numClasses, "resnet18");

%Balance Classes Using Class Weighting

imageFreq = tbl.PixelCount ./ tbl.ImagePixelCount;

classWeights = median(imageFreq) ./ imageFreq;

%Specify the class weights using a pixelClassificationLayer.

%"comment due to error"

pxLayer = pixelClassificationLayer('Name','labels','Classes',tbl.Name,'ClassWeights',classWeights);

lgraph = replaceLayer(lgraph,"classification",pxLayer);

%Select Training Options

% Define validation data.

pximdsVal = pixelLabelImageDatastore(imdsVal,pxdsVal);

% Define training options.

options = trainingOptions('sgdm', ...

'LearnRateSchedule','piecewise',...

'LearnRateDropPeriod',10,...

'LearnRateDropFactor',0.3,...

'Momentum',0.9, ...

'InitialLearnRate',1e-3, ...

'L2Regularization',0.005, ...

'ValidationData',pximdsVal,...

'MaxEpochs',10, ...

'MiniBatchSize',4, ...

'Shuffle','every-epoch', ...

'CheckpointPath', tempdir, ...

'VerboseFrequency',2,...

'Plots','training-progress',...

'ValidationPatience', 4);

%Start Training

pximds = pixelLabelImageDatastore(imdsTrain,pxdsTrain);

%pximds = tall(pximds);

% for preload training

[net, info] = trainNetwork(pximds,lgraph,options);

%Test Network on One Image

I = readimage(imdsTest,3);

C = semanticseg(I, net);

% display the result \

B = labeloverlay(I,C,'Colormap',cmap,'Transparency',0.4);

imshow(B)

pixelLabelColorbar(cmap, classes);

%ompare the results in C with the expected ground truth stored in pxdsTest

expectedResult = readimage(pxdsTest,3);

actual = uint8(C);

expected = uint8(expectedResult);

imshowpair(actual, expected)

% using jecord function

iou = jaccard(C,expectedResult);

table(classes,iou)

%Evaluate Trained Network

pxdsResults = semanticseg(imdsTest,net, ...

'MiniBatchSize',4, ...

'WriteLocation',tempdir, ...

'Verbose',false);

metrics = evaluateSemanticSegmentation(pxdsResults,pxdsTest,'Verbose',false);

metrics.DataSetMetrics

metrics.ClassMetrics

% start all the function one by one

% start of CamVid PixelLabelDs.m

function labelIDs = camvidPixelLabelIDs()

labelIDs = [

% "Sky"

135 206 236; ... % "Sky"

% "Building"

166 227 027; ... % "Bridge"

074 194 154; ... % "Building"

163 046 018; ... % "Wall"

% "Pole"

030 073 204; ... % "Column_Pole"

251 116 077; ... % "TrafficCone"

132 017 209; ... % "electric_pole"

078 080 199; ... % "street light"

023 125 202; ... % "trafic light"

% Road

119 061 128; ... % "pavedRoad"

217 112 230; ... % "unpaved_road"

212 218 030; ... % "solid yellow line"

118 103 122; ... % "pavement_Sidewalk"

225 150 119; ... % "concrete_barrier"

155 104 041; ... % "fence"

% "vehicle"

117 082 016; ... % "police_vehicle "

126 165 140; ... % "constuction_vehicle"

169 166 090; ... % "military_vehicle"

160 237 109; ... % "firebrigade"

074 054 062; ... % "traditional truck"

178 009 055; ... % "container truck"

200 029 177; ... % "Hino_bus"

240 009 145; ... % "traditional bus"

255 001 077; ... % "car"

200 165 038; ... % "van"

080 216 156; ... % "mini_van"

092 083 219; ... % "rickshaw"

253 057 087; ... % "cart"

% 005 172 216; ... % "qingqi"

082 035 004; ... % "pickup"

196 004 086; ... % "mini-pickup"

163 176 081; ... % "tanker"

082 043 112; ... % "Ambulance"

192 099 099; ... % "motor_bike"

% "Tree"

055 088 015; ... % "Tree"

128 222 091; ... % "VegetationMisc"

% "SignSymbol"

196 138 200; ... % "Signboard"

240 146 005; ... % "speedlimit"

211 203 050; ... % "Ad"

181 113 130; ... % "Directionboard"

097 132 151; ... % "distanceboard"

% "Pedestrian"

167 046 047; ... % "Pedestrian"

% 081 058 088; ... % "Animal"

% "mountains"

120 114 104; ...% mountains

001 037 003;... % grassy_road_divider

252 002 002; ...% bicycle

];

end

%end of CamVid PixelLabelDs.m

% start of Pixelcolor bar

function pixelLabelColorbar(cmap, classNames)

% Add a colorbar to the current axis. The colorbar is formatted

% to display the class names with the color.

colormap(gca,cmap)

% Add colorbar to current figure.

c = colorbar('peer', gca);

% Use class names for tick marks.

c.TickLabels = classNames;

numClasses = size(cmap,1);

% Center tick labels.

c.Ticks = 1/(numClasses*2):1/numClasses:1;

% Remove tick mark.

c.TickLength = 0;

end

% end of Pixelcolor bar

% end of Pixelcolor bar

%startcomvidcolormap.m

function cmap = camvidColorMap()

% Define the colormap used by CamVid dataset.

cmap = [

135 206 236; ... % "Sky"

166 227 027; ... % "Bridge"

074 194 154; ... % "Building"

163 046 018; ... % "Wall"

030 073 204; ... % "Column_Pole"

251 116 077; ... % "TrafficCone"

132 017 209; ... % "electric_pole"

078 080 199; ... % "street light"

023 125 202; ... % "trafic light"

119 061 128; ... % "pavedRoad"

217 112 230; ... % "unpaved_road"

212 218 030; ... % "solid yellow line"

118 103 122; ... % "pavement_Sidewalk"

225 150 119; ... % "concrete_barrier"

155 104 041; ... % "fence"

117 082 016; ... % "police_vehicle "

126 165 140; ... % "constuction_vehicle"

169 166 090; ... % "military_vehicle"

160 237 109; ... % "firebrigade"

074 054 062; ... % "traditional truck"

178 009 055; ... % "container truck"

200 029 177; ... % "Hino_bus"

240 009 145; ... % "traditional bus"

255 001 077; ... % "car"

200 165 038; ... % "van"

080 216 156; ... % "mini_van"

092 083 219; ... % "rickshaw"

253 057 087; ... % "cart"

%005 172 216; ... % "qingqi"

082 035 004; ... % "pickup"

196 004 086; ... % "mini-pickup"

163 176 081; ... % "tanker"

082 043 112; ... % "Ambulance"

192 099 099; ... % "motor_bike"

055 088 015; ... % "Tree"

128 222 091; ... % "VegetationMisc"

196 138 200; ... % "Signboard"

240 146 005; ... % "speedlimit"

211 203 050; ... % "Ad"

181 113 130; ... % "Directionboard"

097 132 151; ... % "distanceboard"

167 046 047; ... % "Pedestrian"

% 081 058 088; ... % "Animal"

120 114 104; ... % mountains

001 037 003; ... % grassy_road_divider

252 002 002; ...% bicycle

];

% Normalize between [0 1].

cmap = cmap ./ 255;

end

%end code of Comvid colormap.m

% partition comvid

function [imdsTrain, imdsVal, imdsTest, pxdsTrain, pxdsVal, pxdsTest] = partitionCamVidData(imds,pxds)

% Partition CamVid data by randomly selecting 60% of the data for training. The

% rest is used for testing.

% Set initial random state for example reproducibility.

rng(0);

numFiles = numel(imds.Files);

shuffledIndices = randperm(numFiles);

% Use 60% of the images for training.

numTrain = round(0.80 * numFiles);

trainingIdx = shuffledIndices(1:numTrain);

% Use 20% of the images for validation

numVal = round(0.10 * numFiles);

valIdx = shuffledIndices(numTrain+1:numTrain+numVal);

% Use the rest for testing.

testIdx = shuffledIndices(numTrain+numVal+1:end);

% Create image datastores for training and test.

trainingImages = imds.Files(trainingIdx);

valImages = imds.Files(valIdx);

testImages = imds.Files(testIdx);

imdsTrain = imageDatastore(trainingImages);

imdsVal = imageDatastore(valImages);

imdsTest = imageDatastore(testImages);

% Extract class and label IDs info.

classes = pxds.ClassNames;

labelIDs = camvidPixelLabelIDs();

% Create pixel label datastores for training and test.

trainingLabels = pxds.Files(trainingIdx);

valLabels = pxds.Files(valIdx);

testLabels = pxds.Files(testIdx);

pxdsTrain = pixelLabelDatastore(trainingLabels, classes, labelIDs);

pxdsVal = pixelLabelDatastore(valLabels, classes, labelIDs);

pxdsTest = pixelLabelDatastore(testLabels, classes, labelIDs);

end

% end of partiction camvid

}

[Marya Rasib]

i also try with small batch size 1, and 2 but still have same issue " out of memory"

[Joss Knight]

It does seem as though your system does not have sufficient memory to support training this model, you might have to make it smaller.

[Marya Rasib]

if we reduce the size of the images, what will be the effect on accuracy?

[Joss Knight]

I am not an expert on Deep Learning. My understanding is that for semantic segmentation, you are better off cropping the input than downsampling it. However it is normal for the model's trained input to be smaller than the images it is used to process - the semanticseg function takes care of this.