Error using minibatchqueue using featureInputLayer

Hello everyone,

I'm working on enhancing my neural network by incorporating additional features, similar to the approach described in the MathWorks guide on Training Network on Image and Feature Data.

The existing network, which currently doesn't include these extra features, functions well. It's a straightforward sequence classification model using a wordEmbeddingLayer and BiLSTM layers, mainly for predicting the membership class of entire sequences using sequence data.

However, I'm encountering some challenges when adding an additional feature layer. I would appreciate any advice or insights on how to effectively integrate this layer into my model. If anyone has experience or suggestions to share, it would be of great help!

This is a code without aditional features. And it works perfect (well good enough):

inputSize = 1;
embeddingDimension = 30;
numHiddenUnits = 128;
numWords = enc.NumWords;
numClasses = numel(categories(YTrain));
l2reg = 0.001; 

layers0 = [ ...
    sequenceInputLayer(inputSize)
    wordEmbeddingLayer(embeddingDimension,numWords)
    bilstmLayer(numHiddenUnits,'OutputMode','sequence')
    batchNormalizationLayer
    dropoutLayer(0.2)
    bilstmLayer(numHiddenUnits,'OutputMode','last')
    batchNormalizationLayer
    dropoutLayer(0.2)
    fullyConnectedLayer(64) 
    reluLayer 
    dropoutLayer(0.2) 
    fullyConnectedLayer(32)
    reluLayer 
    fullyConnectedLayer(numClasses, 'WeightLearnRateFactor', 1, 'BiasLearnRateFactor', 1, 'WeightL2Factor', l2reg)
    softmaxLayer
    classificationLayer] 

options0 = trainingOptions('adam', ...
        "MaxEpochs", 200, ...                   
        "MiniBatchSize", 256, ...             
        "ValidationFrequency", 111, ...
        "ExecutionEnvironment", "gpu", ...
        "GradientThreshold", 1, ...
        "SequenceLength", "longest", ...
        "Shuffle","every-epoch", ...
        'ValidationPatience', 30, ... 
        'ValidationData',{SeqVal, YVal}, ...
        'Plots','training-progress', ...
        'Verbose',false);

net0 = trainNetwork(SeqTrain,YTrain,layers0,options0);

However, when I try adding an additional feature layer:

layers1 = [ ...
    sequenceInputLayer(inputSize)
    wordEmbeddingLayer(embeddingDimension,numWords)
    bilstmLayer(numHiddenUnits,'OutputMode','sequence')
    batchNormalizationLayer
    dropoutLayer(0.2)
    bilstmLayer(numHiddenUnits,'OutputMode','last')
    batchNormalizationLayer
    dropoutLayer(0.2)
    concatenationLayer(1,2,Name="cat")
    fullyConnectedLayer(64) 
    reluLayer
    dropoutLayer(0.2) 
    fullyConnectedLayer(32) 
    reluLayer 
    fullyConnectedLayer(numClasses, 'WeightLearnRateFactor', 1, 'BiasLearnRateFactor', 1, 'WeightL2Factor', l2reg)
    softmaxLayer
    classificationLayer] 

lgraph = layerGraph(layers0);
numFeatures=400;
featInput = featureInputLayer(numFeatures,Name="features");
lgraph = addLayers(lgraph,featInput);
lgraph = connectLayers(lgraph,"features","cat/in2");

% Prepare the data for training
X2pTrainm = cell2mat(X2pTrain);
X2pValm = cell2mat(X2pVal);
dsSeqTrain = arrayDatastore(SeqTrain, IterationDimension=2);
dsX2pTrain = arrayDatastore(X2pTrainm);
dsYTrain = arrayDatastore(YTrain);
dsTrain = combine(dsSeqTrain, dsX2pTrain, dsYTrain);
dsSeqVal = arrayDatastore(SeqVal, IterationDimension=2);
dsX2pVal = arrayDatastore(X2pValm);
dsYVal = arrayDatastore(YVal);
dsVal = combine(dsSeqVal, dsX2pVal, dsYVal);

analyzeNetwork(lgraph)

options1 = trainingOptions('adam', ...
        "MaxEpochs", 200, ...                   
        "MiniBatchSize", 256, ...             
        "ValidationFrequency", 111, ...
        "ExecutionEnvironment", "gpu", ...
        "GradientThreshold", 1, ...
        "SequenceLength", "longest", ...
        "Shuffle","every-epoch", ...
        'ValidationPatience', 30, ... 
        'ValidationData', dsVal, ...
        'Plots','training-progress', ...
        'Verbose',false);



% Train the network
net1 = trainNetwork(dsTrain, lgraph, options1);

I got this error:

Error using trainNetwork

Unable to apply function specified by 'MiniBatchFcn' value.

Caused by:

Error using minibatchqueue

Unable to apply function specified by 'MiniBatchFcn' value.

Error using padsequences

Input sequences must be numeric or categorical arrays.

When I analyze the network using analyzeNetwork(lgraph), everything appears to be fine. My sequence data vary in length, but this wasn't an issue for the network without additional features (net0). However, for the network with additional features (net1), it's not working properly. I've tried experimenting by padding the data before training to equalize the sequence lengths, but that hasn't been successful. My additional features are stored in an N*400 matrix, and everything seems okay with it. Does anyone have any suggestions or ideas?

[Matt J]

What about when you view the network in deepNetworkDesigner? Does it all look the way you intend?

[Artur Pokropek]

Everything appears to be in order within the Deep Network Designer. After analyzing the network, no issues were identified. The issue is most likely related to the preparation of the datastore, and the error seems to be associated with this aspect.

This is how im trying to construct datastore for the network using padded sequecnes

dsSeqTrain = arrayDatastore(SeqTrainPad);
dsX2pTrain = arrayDatastore(X2pTrainm);
dsYTrain = arrayDatastore(YTrain);
dsTrain = combine(dsSeqTrain, dsX2pTrain, dsYTrain);

X2pTrainm is a simple matrix of features (56091 x 400) YTrain is a vector for clasyfication (56091 x 400). In SeqTrainPad I have tokenized sequences:

head(SeqTrainPad)
{[1 2 3 4 2 5 6 2 7 8 2 9 10 2 11 12 2 13 14 2 15 16 2 17 18 19 20 2 21 22 2 23 24 2 25 26 2 27 28 29 29 29 2 30 31 2 32 33 2 34 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 … ] (1×400 double)}
{[1 2 3 4 2 5 6 2 7 8 2 9 10 2 11 12 2 13 14 2 15 16 2 17 18 19 20 2 21 22 2 23 24 2 25 26 2 27 28 29 29 29 2 30 31 2 32 33 2 34 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 … ] (1×400 double)}
{[1 2 3 4 2 5 6 2 7 8 2 9 10 2 11 12 2 13 14 2 15 16 2 17 18 19 20 2 21 22 2 23 24 2 25 26 2 27 28 29 29 29 2 30 31 2 32 33 2 34 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 … ] (1×400 double)}
{[1 2 3 4 2 5 6 2 7 8 2 9 10 2 11 12 2 13 14 2 15 16 2 17 18 19 20 2 21 22 2 23 24 2 25 26 2 27 28 29 29 29 2 30 31 2 32 33 2 34 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 … ] (1×400 double)}

I would be grateful for any assistance, as I keep encountering the same error code repeatedly.

Answer 1

[Gayathri]

Hi @Artur Pokropek,

I understand that you are getting the “minibatchqueue” error while adding a “feature” layer into the Neural Network.

I created sample data with the dimensions you specified, and I encountered a similar issue as mentioned in the question. The error says “Input sequences must be numeric or categorical arrays.” For this first we will have to convert the “SeqTrain” and “SeqVal” cell variables to matrix in double format. To perform this, please refer to the code attached below.

convertToNumeric = @(cellArray) cellfun(@(x) double(x), cellArray, 'UniformOutput', false);

% Convert cell arrays to numeric arrays
SeqTrainNumeric = cell2mat(convertToNumeric(SeqTrain));
SeqValNumeric = cell2mat(convertToNumeric(SeqVal));

With the above changes “minibatchqueue” error was solved but it had an additional error pertaining to the input dimension. Please ensure that your “X2pTrainm” has dimensions in the format “numFeatures X numSamples”. And then the input data can be converted to “arrayDatastore” format as mentioned below.

dsSeqTrain = arrayDatastore(SeqTrainNumeric,IterationDimension=1);
dsX2pTrain = arrayDatastore(X2pTrainm,IterationDimension=2);
dsYTrain = arrayDatastore(YTrain);
dsTrain = combine(dsSeqTrain, dsX2pTrain, dsYTrain);
dsSeqVal = arrayDatastore(SeqValNumeric, IterationDimension=1);
dsX2pVal = arrayDatastore(X2pValm,IterationDimension=2);
dsYVal = arrayDatastore(YVal);
dsVal = combine(dsSeqVal, dsX2pVal, dsYVal);

This would prepare the input data into the Neural Network in the expected format. Please ensure that read(dsTrain) produces output as shown below.

While mentioning the “options1” for training “dsVal” has to be mentioned corresponding to “ValidationData” option. This is because the model expects the data to be in “arrayDatastore” format.

options0 = trainingOptions('adam', ...
        "MaxEpochs", 200, ...                   
        "MiniBatchSize", 256, ...             
        "ValidationFrequency", 111, ...
        "GradientThreshold", 1, ...
        "SequenceLength", "longest", ...
        "Shuffle","every-epoch", ...
        'ValidationPatience', 30, ... 
        'ValidationData',dsVal, ...
        'Plots','training-progress', ...
        'Verbose',false);

By incorporating the above changes, we can train the Neural Network with the additional “feature” layer.

For more information on “arrayDatastore” and “Training Network on Image and Feature Data”, please refer to the below links.

https://www.mathworks.com/help/releases/R2023b/matlab/ref/matlab.io.datastore.arraydatastore.html

https://www.mathworks.com/help/releases/R2023b/deeplearning/ug/train-network-on-image-and-feature-data.html

Hope you find this information helpful.