From the MSE graph image shared by you, it seems that the neural netowork is overfitting your training data. Since the whole process of training a neural network for a cusom dataset depends on a lot of heuristics, here are some suggestions that you should try out:
1. Data Preprocessing
- Ensure that your input features (Speed, Compression Ratio, A/F ratio, Load) and target values are on similar scales. Differences in scale can cause the network to learn unevenly or converge slowly.Use functions like mapminmax or zscore for both inputs and targets.
- If you normalize training data, make sure you apply the same normalization parameters to your test data—this keeps predictions consistent with the model training.
2. Data Splitting and Cross-Validation
- A 35/14 split (roughly 70/30) is standard. However, with only 49 samples, any single split might not represent the data variability well. Consider using k-fold cross-validation (or even leave-one-out cross-validation). For example, a 5- or 7-fold cross-validation ensures each data point is used for testing exactly once, giving you a better estimate of model performance on unseen data.
3. Network Architecture and Complexity
- With just 49 data points, using 10 neurons in a hidden layer might be more than necessary, as it can lead to overfitting. Start smaller (e.g., 5 neurons) and see if the performance is comparable.Experiment incrementally by adjusting the number of neurons and layers. Sometimes a simpler network generalizes better when data is scarce.
4. Training Function and Regularization
- While TRAINLM is fast and effective for many problems, it may overfit on very small datasets.Consider trying Bayesian Regularization (trainbr), which adds a regularization term during training. Regularization helps prevent overfitting by penalizing large weights. You might need to adjust the performParam.regularization parameter of the network.
Although nntool is convenient, sometimes scripting with functions like crossvalind (from the Statistics and Machine Learning Toolbox) or by manually dividing your data can offer better control. Refer the script below that implements the above suggestions:
% Load your data (example matrices for illustration)
% X: 4 x 49 matrix containing [Speed; Compression Ratio; A/F ratio; Load]
% T: target data, e.g., 1 x 49 matrix
% Normalize the data
[X, ps_input] = mapminmax(X);
[T, ps_target] = mapminmax(T);
% Create a feedforward network with a smaller hidden layer
hiddenSizes = 5; % Adjust as needed to prevent overfitting
net = feedforwardnet(hiddenSizes, 'trainlm'); % You could try 'trainbr' here
% Specify data division parameters
net.divideFcn = 'dividerand';
net.divideParam.trainRatio = 0.7;
net.divideParam.valRatio = 0.15;
net.divideParam.testRatio = 0.15;
% Optionally, adjust training parameters like epochs if needed
net.trainParam.epochs = 1000;
net.trainParam.max_fail = 6; % For early stopping based on validation performance
% Train the network
[net, tr] = train(net, X, T);
% Test the network
Y = net(X);
Y_original = mapminmax('reverse', Y, ps_target);
% Assess performance using your preferred metrics
perf = perform(net, T, Y);
You can also go through the following MATLAB documentation link: https://www.mathworks.com/help/deeplearning/ug/improve-neural-network-generalization-and-avoid-overfitting.html
