Deep Learning Semantic Segmentation Example

In order to familiarize myself with semantic segmentation and convolutional neural networks I am going through this tutorial by MathWorks:

Semantic Segmentation Using Deep Learning

I did not use the pretrained version of Segnet since I wanted to test on my custom data set. All code is the same, however I have different classes, and **fewer labels**. Below image shows the label name and amount of pixels associated with each.

To make up for the low pixel data for class 2, median frequency balancing was performed.

    imageFreq = tbl.PixelCount ./ tbl.ImagePixelCount
    classWeights = median(imageFreq) ./ imageFreq

I proceed to train the network using the code provided in the example with the `options` and `lgraph` unchanged. The SegNet network is created with weights initialized from the VGG-16 network

Unlike the example, I get a much lower global accuracy:

To gain further insight I plotted the Mini-batch accuracy and Mini-batch loss against each iteration.

It is clearly seen that the accuracy fluctuates wildly and ends up worse than it started, so the network learned absolutely nothing! However the loss decreased gradually.

A possible solution I propose would be to use inverse frequency balancing. However, in the example above, median frequency balancing was already performed, so I doubt how much this would help.

Is the terrible performance related to simply not having enough training data? Can anything be be done to improve performance with existing data?

Any suggestions are greatly appreciated.