Is a Bi-GRU available - bidirectional Gated Recurrent Unit (GRU) - or a way to implement a Bi-GRU?

The following artificial recurrent neural network (RNN) architectures are available:

layer = gruLayer(numHiddenUnits)

layer = lstmLayer(numHiddenUnits)

layer = bilstmLayer(numHiddenUnits)

Whereas, I would like to know if an Bi-GRU exists or can be defined?

Thank you for your help.

Answer 1

[Amanjit Dulai]

A bi-LSTM layer works by applying two LSTM layers on the data; one in the forward direction and one in the reverse direction. You can apply an LSTM function in the reverse direction by flipping the data. The results from these two LSTM layers is then concatenated together to form the output of the bi-LSTM layer. So if we want to implement a bi-GRU layer, we can do this by using a custom flip layer together with GRU layers. A custom flip layer can be implemented as follows:

classdef FlipLayer < nnet.layer.Layer

methods

function layer = FlipLayer(name)

layer.Name = name;

end

function Y = predict(~, X)

Y = flip(X,3);

end

end

end

We can implement a bi-GRU layer with OutputMode="sequence" by arranging layers in the way shown below:

We can implement a bi-GRU layer with with OutputMode="last" by arranging layers in the way shown below:

Below is a short example showing how to use the custom flip layer mentioned above to implement a network with two bi-GRU layers (one with OutputMode="sequence" and oine with OutputMode="last"):

[XTrain, YTrain] = japaneseVowelsTrainData;

lg = layerGraph();

lg = addLayers(lg, [

sequenceInputLayer(12, "Name", "input")

gruLayer( 100, 'OutputMode', 'sequence', ...

"Name", "gru1")

concatenationLayer(1, 2, "Name", "cat1")

gruLayer( 100, 'OutputMode', 'last', ...

"Name", "gru3")

concatenationLayer(1, 2, "Name", "cat2")

fullyConnectedLayer(9)

softmaxLayer

classificationLayer()] );

lg = addLayers( lg, [

FlipLayer("flip1")

gruLayer( 100, 'OutputMode', 'sequence', ...

"Name", "gru2" )

FlipLayer("flip2")] );

lg = addLayers(lg, [

FlipLayer("flip3")

gruLayer( 100, 'OutputMode', 'last', ...

"Name", "gru4" )] );

lg = connectLayers(lg, "input", "flip1");

lg = connectLayers(lg, "flip2", "cat1/in2");

lg = connectLayers(lg, "cat1", "flip3");

lg = connectLayers(lg, "gru4", "cat2/in2");

options = trainingOptions('adam', 'Plots', 'training-progress');

net = trainNetwork(XTrain, YTrain, lg, options);

[XTest, YTest] = japaneseVowelsTestData;

YPred = classify(net, XTest);

accuracy = sum(YTest == YPred)/numel(YTest)

[Ronny Guendel]

Thank you for your detailed description. It is very helpful.

I was wondering if the example code requres additional toolboxes and I am using Matlab version 2021a.

It gives me the following error:

Error using

nnet.internal.cnn.util.validateLayersForLayerGraph>iAssertUniqueAndNonEmptyLayerNames

(line 45)

Layer names in layer array must be nonempty.

Error in nnet.internal.cnn.util.validateLayersForLayerGraph (line 33)

iAssertUniqueAndNonEmptyLayerNames(larray, existingLayers);

Error in nnet.cnn.LayerGraph>iValidateLayers (line 550)

larray = nnet.internal.cnn.util.validateLayersForLayerGraph(larray,

existingLayers);

Error in nnet.cnn.LayerGraph/addLayers (line 184)

larray = iValidateLayers(larray, existingLayers);

Error in biGRU (line 3)

lg = addLayers(lg, [

[Ronny Guendel]

I found the issue:

The names for the layers must be not empty such as,

fullyConnectedLayer(9, 'Name', 'fc')

softmaxLayer('Name', 'softmax')

classificationLayer('Name', 'classoutput')

The full working code:

[XTrain, YTrain] = japaneseVowelsTrainData;

lg = layerGraph();

lg = addLayers(lg, [

sequenceInputLayer(12, "Name", "input")

gruLayer( 100, 'OutputMode', 'sequence', ...

"Name", "gru1")

concatenationLayer(1, 2, "Name", "cat1")

gruLayer( 100, 'OutputMode', 'last', ...

"Name", "gru3")

concatenationLayer(1, 2, "Name", "cat2")

fullyConnectedLayer(9, 'Name', 'fc')

softmaxLayer('Name', 'softmax')

classificationLayer('Name', 'classoutput')] );

lg = addLayers( lg, [

FlipLayer("flip1")

gruLayer( 100, 'OutputMode', 'sequence', ...

"Name", "gru2" )

FlipLayer("flip2")] );

lg = addLayers(lg, [

FlipLayer("flip3")

gruLayer( 100, 'OutputMode', 'last', ...

"Name", "gru4" )] );

lg = connectLayers(lg, "input", "flip1");

lg = connectLayers(lg, "flip2", "cat1/in2");

lg = connectLayers(lg, "cat1", "flip3");

lg = connectLayers(lg, "gru4", "cat2/in2");

options = trainingOptions('adam', 'Plots', 'training-progress');

net = trainNetwork(XTrain, YTrain, lg, options);

[XTest, YTest] = japaneseVowelsTestData;

YPred = classify(net, XTest);

accuracy = sum(YTest == YPred)/numel(YTest)

[yj z]

Thank you for your detailed description. It is very helpful.

I was wondering if the example code requres additional toolboxes and I am using Matlab version 2020b.

It gives me the following error:

Undefined function 'FlipLayer' for input arguments of type 'string'.

[Amanjit Dulai]

That error seems to indicate that FlipLayer can not be found on the path. You need to copy the code in the answer that begins with "classdef FlipLayer ..." into a file called "FlipLayer.m", and make sure it is in your current working directory (or anywhere else that is on the MATLAB path).

[luolaijng]

Thank you very much for your detailed explanation

[Farnam Hojatpanah]

I Have a question. Why did you flip the input along with the third dimension?

[Amanjit Dulai]

The third dimension is the time dimension. For a bi-GRU, we want one GRU layer to operate on the sequence in the forward time direction, and then we want one GRU layer to operate on the sequence in the reverse time direction. We can get it to operate on the sequence in the reverse time direction by flipping in the third dimension.

Answer 2

[Ronny Guendel]

The full working code for me:

[XTrain, YTrain] = japaneseVowelsTrainData;

lg = layerGraph();

lg = addLayers(lg, [

sequenceInputLayer(12, "Name", "input")

gruLayer( 100, 'OutputMode', 'sequence', ...

"Name", "gru1")

concatenationLayer(1, 2, "Name", "cat1")

gruLayer( 100, 'OutputMode', 'last', ...

"Name", "gru3")

concatenationLayer(1, 2, "Name", "cat2")

fullyConnectedLayer(9, 'Name', 'fc')

softmaxLayer('Name', 'softmax')

classificationLayer('Name', 'classoutput')] );

lg = addLayers( lg, [

FlipLayer("flip1")

gruLayer( 100, 'OutputMode', 'sequence', ...

"Name", "gru2" )

FlipLayer("flip2")] );

lg = addLayers(lg, [

FlipLayer("flip3")

gruLayer( 100, 'OutputMode', 'last', ...

"Name", "gru4" )] );

lg = connectLayers(lg, "input", "flip1");

lg = connectLayers(lg, "flip2", "cat1/in2");

lg = connectLayers(lg, "cat1", "flip3");

lg = connectLayers(lg, "gru4", "cat2/in2");

options = trainingOptions('adam', 'Plots', 'training-progress');

net = trainNetwork(XTrain, YTrain, lg, options);

[XTest, YTest] = japaneseVowelsTestData;

YPred = classify(net, XTest);

accuracy = sum(YTest == YPred)/numel(YTest)

Answer 3

[义]

why i can't find the FlipLayer in matlab2022a?

[Amanjit Dulai]

FlipLayer is a custom layer you need to implement. The code is shown below:

classdef FlipLayer < nnet.layer.Layer

methods

function layer = FlipLayer(name)

layer.Name = name;

end

function Y = predict(~, X)

Y = flip(X,3);

end

end

end

[义]

If my data is a 1*n power system load data, how should I set this parameter?

Answer 4

[义]

If my data is a 1*n power system load data, how should I set this parameter?

Answer 5

[song]

If my data is a 1*n bearing system load data, how should I set this parameter?

Answer 6

[Artem Lensky]

Below is a multiblock BiGRU implementation with layer normalization and dropout layers.

The models takes the following parameters

numFeatures, numBlocks, numHiddenUnits, numResponse, dropoutFactor

and the function constructing BiGRU is implemented as follows:

function lgraph = constructBiGRU(numFeatures,numBlocks,numHiddenUnits,numResponse,dropoutFactor)
    arguments
        numFeatures    = 1,
        numBlocks      = 1,
        numHiddenUnits = 32,
        numResponse    = 3,
        dropoutFactor  = 0,
    end
    
    layer  = [sequenceInputLayer(numFeatures,'Name','sequenceInputLayer')];
    lgraph = layerGraph(layer);
    outputName = lgraph.Layers(end).Name;
    
    for i = 1:numBlocks
        layers = [gruLayer(numHiddenUnits, OutputMode='sequence',Name="gru_1_" + i)
                  concatenationLayer(1, 2, Name="cat_" + i)
                  dropoutLayer(dropoutFactor, Name="dropout"+ i)
                  layerNormalizationLayer(Name="layernorm_" + i)];
        lgraph = addLayers(lgraph, layers);

        layers = [FlipLayer("flip_1_" + i)
                  gruLayer(numHiddenUnits, OutputMode='sequence',Name="gru_2_" + i)
                  FlipLayer("flip_2_" + i)];

        lgraph = addLayers(lgraph, layers);
        lgraph = connectLayers(lgraph, outputName, "gru_1_" + i);
        lgraph = connectLayers(lgraph, outputName, "flip_1_" + i);
        lgraph = connectLayers(lgraph, "flip_2_" + i, "cat_" + i + "/in2"); 
        
        outputName = "layernorm_" + i;
    end
    % Remove last dropout layer. In case the network contains a single
    % BiGRU block, no dropout layers will be added at all.
    lgraph = removeLayers(lgraph, "layernorm_" + i);

    layers = [fullyConnectedLayer(numResponse,'Name','fc')
              softmaxLayer
              classificationLayer];                
    
    lgraph = addLayers(lgraph, layers);
    lgraph = connectLayers(lgraph, "dropout" + i, "fc");
end