Why does the validation loss first decrease and then increase after about 7 epochs while the training loss keeps decreasing？

Hello everyone

I am using maskrcnn and trainMaskRCNN provided by Matlab to train about 4500 tree crown images. The validation loss first decrease and then increase after about 7 epochs while the training set loss keeps decreasing. I konw it is ''overfitting'' , but i can not fix it out.Can someone give me some advice?

%% Configure Mask R-CNN Network
trainClassNames = ["akamatsu", "karamatsu", "sugi", "hinoki"];
imageSizeTrain = [384 384 3];
net = maskrcnn("resnet50-coco",trainClassNames,InputSize=imageSizeTrain);
%% Train Network
options = trainingOptions("sgdm", ...
    InitialLearnRate=0.001, ...
    LearnRateSchedule="piecewise", ...
    LearnRateDropPeriod=1, ...
    LearnRateDropFactor=0.95, ...
    Plot="training-progress", ...
    Momentum=0.9, ...
    MaxEpochs=30, ...
    MiniBatchSize=2, ...
    L2Regularization=0.0001, ...
    BatchNormalizationStatistics="moving", ...
    ResetInputNormalization=false, ...
    ExecutionEnvironment="gpu", ...
    VerboseFrequency=50, ...
    ValidationData = ds_valid, ...
    ValidationFrequency = 50, ...
    ValidationPatience = Inf);
doTraining = true;
if doTraining
    [net,info] = trainMaskRCNN(ds_train,net,options,FreezeSubNetwork="none");
    save("RGB_1204_30epoch_none.mat","net","info");
end

Answer 1

[Mike Croucher]

I think you are right, this is an example of overfitting. There is no guarenteed way to fix it but there are several things you can try

1. L2 Regularization: You already have a small L2 regularization term (0.0001). Try increasing it slightly (e.g., 0.001 or 0.01) to penalize overly complex models. I suggest using something like Experiment manager to help you find the best value for this (and other) hyperparameters https://uk.mathworks.com/help/stats/experimentmanager-app.html
2. Data Augmentation Increase the variety in your training dataset using data augmentation techniques such as random rotations, flips, scaling, color jittering, and cropping. Use the imageDataAugmenter in MATLAB to apply augmentations dynamically during training. More details at Preprocess Images for Deep Learning
3. Learning Rate Adjustments and batch size Try different schemes and, again, use Experiment manager to loop over a bunch of these hyperparameters.
4. Reduce Model Complexity You are using a complex network on a small dataset. Could you use something simpler?
5. Early Stopping. It could be that the best you are going to be able to do is at the the inflection point where validation loss starts increasing. In which case, stop at at that point and use that model. To do this, see Stop Training Early Using Metrics shown in trainingOptions

Hope these help! There's also a MathWorks article on overfitting at Overfitting - MATLAB & Simulink