Using transformer neural network for classification task

numChannels = inputSize;
maxPosition = 256;
numHeads = 4;
numKeyChannels = numHeads*32;

layers = [ 
        sequenceInputLayer(numChannels,Name="input")
        positionEmbeddingLayer(numChannels, maxPosition, Name="pos-emb");
        additionLayer(2, Name="add")
        selfAttentionLayer(numHeads,numKeyChannels,'AttentionMask','causal')
        selfAttentionLayer(numHeads,numKeyChannels)
        indexing1dLayer("last")
        fullyConnectedLayer(numClasses)
        softmaxLayer
        classificationLayer];

lgraph = layerGraph(layers);
lgraph = connectLayers(lgraph, "input", "add/in2");

maxEpochs = 100;
miniBatchSize = 32;
learningRate = 0.001;
solver = 'adam';
shuffle = 'every-epoch';
gradientThreshold = 10;
executionEnvironment = "auto"; % chooses local GPU if available, otherwise CPU

options = trainingOptions(solver, ...
                          'Plots','training-progress', ...
                          'MaxEpochs', maxEpochs, ...
                          'MiniBatchSize', miniBatchSize, ...
                          'Shuffle', shuffle, ...
                          'InitialLearnRate', learningRate, ...
                          'GradientThreshold', gradientThreshold, ...
                          'ExecutionEnvironment', executionEnvironment);

The input size is 12, so there are 12 features.

numClasses is 4, so I am classifying it into 4 class.

But it gives the following error when I try to run it

"

Error in test123_20240727 (line 195)

net=trainNetwork(XTrain, YTrain, layers, options);

Caused by:

Layer 'add': Unconnected input. Each layer input must be connected to the output of another layer.

"

line 195 is "net=trainNetwork(XTrain, YTrain, layers, options);"

Can anyone help me with this?

[Umar]

Hi @ haohaoxuexi1,

By connecting the output of the 'pos-emb' layer to the input of the 'add' layer using connectLayers, and making sure that all layer inputs are properly linked, should resolve the unconnected input error. Here is the updated code.

numChannels = 12;

maxPosition = 256;

numHeads = 4;

numKeyChannels = numHeads * 32;

layers = [

    sequenceInputLayer(numChannels, 'Name', 'input')

    positionEmbeddingLayer(numChannels, maxPosition, 'Name', 'pos-emb')

    additionLayer(2, 'Name', 'add')

    selfAttentionLayer(numHeads, numKeyChannels, 'AttentionMask', 

'causal')

    selfAttentionLayer(numHeads, numKeyChannels)

    indexing1dLayer('last')

    fullyConnectedLayer(4)

    softmaxLayer

    classificationLayer];

lgraph = layerGraph(layers);

lgraph = connectLayers(lgraph, 'pos-emb', 'add/in2'); % Connect 'pos-emb'

output to 'add' input

maxEpochs = 100;

miniBatchSize = 32;

learningRate = 0.001;

solver = 'adam';

shuffle = 'every-epoch';

gradientThreshold = 10;

executionEnvironment = 'auto'; % chooses local GPU if available, otherwise CPU

options = trainingOptions(solver, ...

    'Plots', 'training-progress', ...

    'MaxEpochs', maxEpochs, ...

    'MiniBatchSize', miniBatchSize, ...

    'Shuffle', shuffle, ...

    'InitialLearnRate', learningRate, ...

    'GradientThreshold', gradientThreshold, ...

    'ExecutionEnvironment', executionEnvironment);

% Assuming XTrain and YTrain are your training data

net = trainNetwork(XTrain, YTrain, lgraph, options); % Use lgraph instead of layers

Hope this should help resolve your problem.

[haohaoxuexi1]

@Umar

thank you Umar,

"net = trainNetwork(XTrain, YTrain, lgraph, options); % Use lgraph instead of layers"

here is the problem, now it works

[Umar]

Hi haohaoxuexi1,

If your problem is resolved, please accept answer by clicking it.

[Matt J]

@Umar You have posted your solution as a comment rather than an answer. Therefore, it is not possible for the OP to Accept-click it.

[Umar]

@Matt J,

For some reason when I click Answer this question, it defaults to draft.

[Umar]

Hi @ haohaoxuexi1,

If you are still having issues with modifying your code, please let us know. We will be happy to help you out.

[haohaoxuexi1]

@Umar Hi Umar, I am good at the moment. Will let u know if I have further question.

Answer 1

[Joss Knight]

You've passed layers instead of lgraph to trainNetwork.

[Umar]

@Joss Knight, Thanks for jumping in. Please advice how to use lgraph to trainNetwork by providing code snippet. Again, thanks for your cooperation.

[Joss Knight]

net=trainNetwork(XTrain, YTrain, lgraph, options);

instead of

net=trainNetwork(XTrain, YTrain, layers, options);