Equal outputs using customized regression output layer

Hello everyone. I defined a customized regression output layer using the provided template and used the MSE for the Loss function:

     classdef mseRegressionLayer < nnet.layer.RegressionLayer
    
    
    methods
        function layer = mseRegressionLayer(name)
            
			
            % Set layer name.
            layer.Name = name;

            % Set layer description.
            layer.Description = 'Mean square error';
        end
        
        function loss = forwardLoss(layer, Y, T)
         
            % the predictions Y and the training targets T.
           
            
            
            
            loss= mse(Y,T,'DataFormat','T')/size(Y,2);
          
            
        end
    end
     end

Then I used this script to a trial dataset of arrays, the same one that has been trained in the example of training shallow neural networks here https://it.mathworks.com/help/deeplearning/ug/train-and-apply-multilayer-neural-networks.html , but now I wanted to build myself the Loss function, so I tried with this:

   layer = mseRegressionLayer('mse');

load bodyfat_dataset % dataset with bodyfatInputs and bodyfatTargets



layers = [
    sequenceInputLayer(13)
    lstmLayer(100,'OutputMode',"last") % Output the last time step of the sequence
    fullyConnectedLayer(1)
    mseRegressionLayer('mse')];


%options = trainingOptions('sgdm');
options = trainingOptions('adam', ...
    'MaxEpochs',100, ...
    'GradientThreshold',1, ...
    'InitialLearnRate',0.001, ...
    'LearnRateSchedule','piecewise', ...
    'LearnRateDropPeriod',100, ...
    'LearnRateDropFactor',0.1,...
    'Verbose',1, ...
    'Plots','training-progress');

N = 100; %number of sequences
cellArrTrain = cell(N,1);
for i = 1:N
    seq = bodyfatInputs(:,i);
    seq = num2cell(seq,1);
    cellArrTrain(i) = seq;
end

net = trainNetwork(cellArrTrain,bodyfatTargets(1:N)',layers,options);


YPred = predict(net,cellArrTrain);
predictionError = YPred - bodyfatTargets;

The problem is that the outputs in this way are all basically the same number... I don't think that is caused by the particular loss function because also using the mae error, on the standard template provided in the documentation, it does the same. The mini batch Loss acceptable, but outputs are too similiar. What can be the reason for it?

Thank you for your help!

Answer 1

[Shashank Gupta]

Hi Fabrizio,

On the first look, the code does look good and it should work, did you check increasing the number of epoch and decreasing the learning rate decay, it seems like, it should converge but learning rate decay is super high which is making loss to converge at higher value than expected and the model is not able to learn. Try removing the decay and check if the training is smooth. I can't find any flag anywhere in the code atleast at the first look.

I hope my suggestion helps you out, Update me if you find anything interesting.

Cheers.