Semantic Segmentation on time series data using Sentinel 1 data: Error Using Trainnet()

I need some product support on using the trainnet() function. Based on documentation, I'm expecting to be able to feed trainnet() an argument of double(SxSxCxB) as a Predictor and type catagorical(SxSxCxB). Instead, I'm encountering an size mismatch error between the network and the data. Some details to reproduce the error are below. Thanks for your support!

Here is some bare bones code that generates the unexpected error:

Image = zeros(256, 256, 1, 5656); % Predictor
Label = zeros(256, 256, 1, 5656); % Target

This yields dimensions of [256 256 1 5656] for each array.

Then the arrays are converted to an argument for the input layer.

Image = dlarray(Image,"SSCB"); % Preferred "SSCBT", but may not be necessary
Label = categorical(dlarray(Label, "SSCB"));

A Bare Bones Network is defined for demonstration:

inputLayer = imageInputLayer([256, 256, 1], Name = 'input',Normalization = 'none'); 
% Convolutional Layer 1: 3x3 kernel, 32 filters, stride 1, padding 1
conv1 = convolution2dLayer(3, 32, 'Stride', 1, 'Padding', 1, 'Name', 'conv1');
batchNorm1 = batchNormalizationLayer('Name', 'bn1');
relu1 = reluLayer('Name', 'relu1');
% Max-pooling Layer 1: 2x2 pooling with stride 2
maxPool1 = maxPooling2dLayer(2, 'Stride', 2, 'Name', 'maxPool1');
% Convolutional Layer 2: 3x3 kernel, 64 filters, stride 1, padding 1
conv2 = convolution2dLayer(3, 64, 'Stride', 1, 'Padding', 1, 'Name', 'conv2');
batchNorm2 = batchNormalizationLayer('Name', 'bn2');
relu2 = reluLayer('Name', 'relu2');
% Max-pooling Layer 2: 2x2 pooling with stride 2
maxPool2 = maxPooling2dLayer(2, 'Stride', 2, 'Name', 'maxPool2');
% Fully Connected Layer: 256 units
fc1 = fullyConnectedLayer(256, 'Name', 'fc1');
relu3 = reluLayer('Name', 'relu3');
% Fully Connected Layer for classification: 10 output classes
fc2 = fullyConnectedLayer(10, 'Name', 'fc2');
% Softmax Layer: Converts the final output into class probabilities
softmaxLayer = softmaxLayer('Name', 'softmax');
% Classification Layer: The output is compared with the true labels
classificationLayer = classificationLayer('Name', 'classOutput');
% Combine all layers into the network
layers = [
    inputLayer
    conv1
    batchNorm1
    relu1
    maxPool1
    conv2
    batchNorm2
    relu2
    maxPool2
    fc1
    relu3
    fc2
    softmaxLayer
    % classificationLayer
    ];

We can then define and observe the network:

%% Define the network
clear inputLayer batchNorm1 batchNorm2 classificationLayer conv1 conv2 fc1 fc2 maxPool1 maxPool2 relu1 relu2 relu3 softmaxLayer
bbnet = dlnetwork(layers);
analyzeNetwork(bbnet)

Error using analyzeNetwork (line 26)
This functionality is not available on remote platforms.

%% Train the network
options = trainingOptions('adam', 'MaxEpochs', 5, 'MiniBatchSize', 64, 'Plots', 'training-progress');
lossFcn = "crossentropy";
net = trainnet(Image, Label, bbnet, lossFcn, options);

Error using trainnet (line 54)

Number of observations in predictors (1) and targets (256) must match. Check that the data and network are

consistent.

Error in BareBonesDLConcepts (line 4)

net = trainnet(Image, Label, bbnet, lossFcn, options);

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The documentation around the trainnet() function makes me think that this should work. Any thoughts?

[David]

Here is the deep learning network analyzer output.

Answer 1

[Gayathri]

Hello @David Hill,

I understand that you need to train a network where both "Image" and "Label" are of the same size. In that case, I would recommend you to use a U-Net architecture. U-Net is particularly well-suited for tasks like image segmentation, where the output prediction must match the size of the input image.

Please refer to the below code for the architecture of the network.

% Encoder part
layers = [
    imageInputLayer([256 256 1], 'Name', 'input', 'Normalization', 'none')
    % Encoder
    convolution2dLayer(3, 32, 'Padding', 'same', 'Stride', 1, 'Name', 'conv1')
    reluLayer('Name', 'relu1')
    maxPooling2dLayer(2, 'Stride', 2, 'Name', 'maxpool1')
    convolution2dLayer(3, 64, 'Padding', 'same', 'Stride', 1, 'Name', 'conv2')
    reluLayer('Name', 'relu2')
    maxPooling2dLayer(2, 'Stride', 2, 'Name', 'maxpool2')
    % Bottleneck
    convolution2dLayer(3, 128, 'Padding', 'same', 'Stride', 1, 'Name', 'bottleneck')
    reluLayer('Name', 'relu_bottleneck')
    % Decoder
    transposedConv2dLayer(3, 64, 'Stride', 2, 'Cropping', 'same', 'Name', 'upconv1')
    reluLayer('Name', 'relu3')
    transposedConv2dLayer(3, 32, 'Stride', 2, 'Cropping', 'same', 'Name', 'upconv2')
    reluLayer('Name', 'relu4')
    transposedConv2dLayer(3, 1, 'Stride', 1, 'Cropping', 'same', 'Name', 'upconv3')
    sigmoidLayer('Name', 'sigmoid_output') % To ensure output is within [0, 1]
];

With this architecture, the training process will start. I have attached a screenshot of the same.

You can also utilize the "unet" function to construct a U-Net architecture. For more details, please refer to the link provided below.

https://www.mathworks.com/help/vision/ref/unet.html

You can also use DeepLab v3+ network, which allows the network to capture multi-scale contextual information without losing resolution. This is particularly useful for segmenting objects at various scales.

For more information on how to train a segmentation model using “deeplabv3plus” refer to the following link.

https://www.mathworks.com/help/vision/ref/deeplabv3plus.html

Hope you find this information helpful!

[David]

Thanks, That explains the function better.

Could you walk me through the "unet" function.

The network will not train given the code below.

Image = zeros(256, 256, 1, 5656); % Predictor
Label = zeros(256, 256, 1, 5656); % Target

I based on my current understanding the inputs are:

Predictor - dlarray range from 0 to 1

target - catagorical array ranges from classes 1 to 5

Image = dlarray(Image,"SSCB"); 
Label = categorical(Label,1:5, ["class 1", "class 2", "class 3", "class 4", "class 5"]);

[U_net, net_Out_Size] = unet([256 256],5);
disp(net_Out_Size)
analyzeNetwork(U_net)

options = trainingOptions('adam', 'MaxEpochs', 5, 'MiniBatchSize', 64, 'Plots', 'training-progress');
lossFcn = "crossentropy";
net = trainnet(Image, Label, U_net, lossFcn, options);

The error I am getting is:

>>

Invalid use of operator.

Iteration    Epoch    TimeElapsed    LearnRate    TrainingLoss
_________    _____    ___________    _________    ____________
Training stopped: Training loss is NaN
>> 

followup questions:

Is it true that a Batch in the "dlarray" function is not the same as the MiniBatchSize in the "trainingOptions" function?

what does Batch in the "dlarray" mean, is it interchangable with Channel/Color?

[Gayathri]

The error you are facing is because "Labels" is not defined properly. The "Label" should be integers if they are being converted to categorical values. Also, make sure that the "Label" values fall within the specified classes.

You can define "Label" as shown below.

Label = randi([1, 5], 256, 256, 1, 5656); % Random integer labels
Label = categorical(Label, 1:5, ["class 1", "class 2", "class 3", "class 4", "class 5"]);

With this change the training will commence without any errors as shown below.

Also to answer your question on "Batch" in "dlarray", it represents the number of observations. Yes, it is different from 'MiniBatchSize" in the "trainingOptions" function. But it is not intercangeable with Channel/Color. It represents the number of observations in the input data.

[David]

In hindsight the title of this thread is misleading and it should be changed to:

Classes with zero instances in the training data.

It seems as though each class must occur at least once

-at least one pixel in one image-.

Somewhere inside the "trainnet" function there was 0/0 = NaN and this prevented training.

The resulted was NaN for initial class weights.

The best practice is probably to only build classes into the network if they are represented in the training.

(I doubt there is a technique to carry dormant classes in a network.)

I would recommend a page with further discussion on training data, perhaps:

Training data - what to watch out for

Deep Learning Training data: The Good the bad and the Ugly

No Classification without Representation (Deep Learning)