Training neural network to solve non-linear equation with multiple input

This is the code that I wrote to train a non-linear equation using deep learning tool box. Notice how I had to split the equation in the begining to help them go through the layers and options without error. This does not work everytime because you cannot split a non-linear equation everytime into an equation of each individual term. For example (X-Y)^2 = X^2 + Y^2 - 2*X*Y where the 2*X*Y can't be split into individual terms that contain only X or Y.

%% Define function to be approximated
fnc1 = @(x1) x1.^5;
fnc2 = @(x2) x2.^3;
fnc =@(x1,x2) x1.^5 + x2.^3;

%% Generate datasets for training and testing
numData = 100;
N_Data_train=3;
N_Data_validation = N_Data_train + 4;

X1Data = linspace(-2,2,numData)';
X2Data = linspace(-3,3,numData)';
XData = [X1Data,X2Data];
Y1Data = fnc1(X1Data);
Y2Data = fnc2(X2Data);
YData = fnc(X1Data,X2Data);
XData1 = {X1Data;X2Data};

%% Define the training data
X1Train = X1Data(1:N_Data_train:end);
X2Train = X2Data(1:N_Data_train:end);
XTrain = [X1Train,X2Train];
Y1Train = fnc1(X1Train);
Y2Train = fnc2(X2Train);
YTrain = fnc(X1Train,X2Train);

X1Validation = X1Data(1:N_Data_validation:end);
X2Validation = X2Data(1:N_Data_validation:end);
XValidation = [X1Validation,X2Validation];
XValidation1 = {X1Validation;X2Validation};
Y1Validation = fnc1(X1Validation);
Y2Validation = fnc2(X2Validation);
YValidation = fnc(X1Validation,X2Validation);

validationData1 = {X1Validation,Y1Validation};
validationData2 = {X2Validation,Y2Validation};
validationData = {XValidation1,YValidation};
validationFrequency = 25; 

%% Define a layer architecture of network

NumNeurons = 50;
N_input1 =size(X1Data,2);  
N_output1=size(Y2Data,2); 
layers1 = [ ...
    featureInputLayer(N_input1, "Name", "Input Layer", 'Normalization','rescale-symmetric')
    fullyConnectedLayer(NumNeurons, "Name", "Fully Connected Layer1of1")
    functionLayer(fnc1, 'Name', 'function1')
    fullyConnectedLayer(N_output1, "Name", "Fully Connected Layer2of1") 
    regressionLayer("Name", "Regression (output) Layer1")
];

NumNeurons = 40;
N_input2 =size(X2Data,2);  
N_output2=size(Y2Data,2); 
layers2 = [ ...
    featureInputLayer(N_input2, "Name", "Input Layer 2", 'Normalization','rescale-symmetric')
    fullyConnectedLayer(NumNeurons, "Name", "Fully Connected Layer1of2")
    functionLayer(fnc2, 'Name','Function2')
    fullyConnectedLayer(N_output2, "Name", "Fully Connected Layer 2of2") 
    regressionLayer("Name", "Regression (output) Layer2")
];

%% Define options for the training
MaxEp = 15000; %20000
LRin =0.001;


Ndata_miniBatch1 = floor(size(X1Train,1)/1); %# of samples in each mini-batch => will take size(XTrain,2)/Ndata_miniBatch iterations with Ndata_miniBatch samples per mini-batch to complete 1 epoch (full pass on the dataset).
opts1 = trainingOptions('adam', ...
    'MaxEpochs',MaxEp, ...
    'InitialLearnRate',LRin,...
    'Shuffle','every-epoch', ...
    'ValidationData',{X1Validation,Y1Validation}, 'ValidationFrequency',validationFrequency, ... 
    'Plots','training-progress', ...
    'MiniBatchSize', Ndata_miniBatch1, ...
    'Verbose',true);

Ndata_miniBatch2 = floor(size(X2Train,1)/1); %# of samples in each mini-batch => will take size(XTrain,2)/Ndata_miniBatch iterations with Ndata_miniBatch samples per mini-batch to complete 1 epoch (full pass on the dataset).
opts2 = trainingOptions('adam', ...
    'MaxEpochs',MaxEp, ...
    'InitialLearnRate',LRin,...
    'Shuffle','every-epoch', ...
    'ValidationData',{X2Validation,Y2Validation}, 'ValidationFrequency',validationFrequency, ... 
    'Plots','training-progress', ...
    'MiniBatchSize', Ndata_miniBatch2, ...
    'Verbose',true);


%% Training the network
[trainedNet1, info1] = trainNetwork(X1Train, Y1Train, layers1, opts1);
[trainedNet2, info2] = trainNetwork(X2Train, Y2Train, layers2, opts2);

But as I said earlier you can't do it without splitting the equation. How to use matlab layers to train the network or is there any way to run the below code without defining custom layers of matlab? There is the error of unconnected layers.

%% Define function to be approximated

fnc =@(x1,x2) (x1 + x2).^2;



%% Generate datasets for training and testing
numData = 100;
N_Data_train=3;
N_Data_validation = N_Data_train + 4;

X1Data = linspace(-2,2,numData)';
X2Data = linspace(-3,3,numData)';
XData = [X1Data,X2Data];
YData = fnc(X1Data,X2Data);

%% Define the training data
X1Train = X1Data(1:N_Data_train:end);
X2Train = X2Data(1:N_Data_train:end);
XTrain = [X1Train,X2Train];
YTrain = fnc(X1Train,X2Train);

X1Validation = X1Data(1:N_Data_validation:end);
X2Validation = X2Data(1:N_Data_validation:end);
XValidation = [X1Validation,X2Validation];
YValidation = fnc(X1Validation,X2Validation);


validationData = {XValidation,YValidation};
validationFrequency = 25; 

%% Define a layer architecture of network

NumNeurons = 50;
N_input1 =size(XData,2);  
N_output1=size(YData,2); 
layers = [ ...
    featureInputLayer(N_input1, "Name", "Input Layer", 'Normalization','rescale-symmetric')
    fullyConnectedLayer(NumNeurons, "Name", "Fully Connected Layer1of1")
    functionLayer(fnc, 'Name', 'function1')
    fullyConnectedLayer(N_output1, "Name", "Fully Connected Layer2of1") 
    regressionLayer("Name", "Regression (output) Layer1")
];



%% Define options for the training
MaxEp = 15000;
LRin =0.001;


Ndata_miniBatch1 = floor(size(X1Train,1)/1);
opts = trainingOptions('adam', ...
    'MaxEpochs',MaxEp, ...
    'InitialLearnRate',LRin,...
    'Shuffle','every-epoch', ...
    'ValidationData',{XValidation,YValidation}, 'ValidationFrequency',validationFrequency, ... 
    'Plots','training-progress', ...
    'MiniBatchSize', Ndata_miniBatch1, ...
    'Verbose',true);



%% Training the network
[trainedNet1, info1] = trainNetwork(XTrain, YTrain, layers, opts);

Answer 1

[Ben]

Hi Fahad,

You are getting the unconnect layers error for the following reason. When you define fnc=@(x1,x2) (x1 + x2).^2 it is a function with 2 inputs. Correspondingly the layer:

functionLayer(fnc,"Name","function1")

has 2 input connections - one for x1 and one for x2.

There are 2 options you could try:

1. Create a layerGraph with 2 input layers, one for x1 and one for x2, and connect these accordingly to the function layer. This example demonstrates training a multiple input network. In this case you would keep X1Data and X2Data as separate inputs into trainNetwork, or combine them with a datastore.
2. Split up the channel dimension inside the network with another function layer that has 2 outputs. For example after the featureInputLayer your XData will have been permuted and normalized, and will have size 2x100. You can then have a layer that splits this into 2 arrays of size 1x100, for example:

splitLayer = functionLayer(@(x) deal(x(1,:),x(2,:)), "Name","split","OuptutNames",["x1","x2"]).

Note that splitLayer would still need to be connected into a layer graph.

Of these options, 1. is the more common approach in our documentation and examples as it is usually easier.

By the way, if you are trying to approximate a function fnc with a neural network, you typically would not want to include fnc directly in the network - e.g. in your code above the fullyConnectedLayer-s could simply learn to "do nothing" to the input data XData (or X1Data and X2Data) and the network would get a loss of 0 because the function it needs to approximate is already in the network definition.

Hope that helps,

Ben