Why does layerNormalizationLayer in Deep Learning Toolbox include T dimension into the batch?

1 ビュー (過去 30 日間)
While implementing a ViT transformer in Matlab, I found at that the layerNormalizationLayer does include the T dimension in the statistics calculated for each sample in the batch. This is problematics when implementing a transformer, since tokens correspond to the T dimension and reference implementations calculate the statistics separately for each token.


John Smith
John Smith 2023 年 3 月 24 日
It seems Mathworks have listened and changed the behavior of layerNormalizationLayer in R2023a.:
Starting in R2023a, by default, the layer normalizes sequence data over the channel and spatial dimensions. In previous versions, the software normalizes over all dimensions except for the batch dimension (the spatial, time, and channel dimensions). Normalization over the channel and spatial dimensions is usually better suited for this type of data. To reproduce the previous behavior, set OperationDimension to "batch-excluded".

その他の回答 (1 件)

Matt J
Matt J 2023 年 3 月 13 日
Perhaps you can fold your T dimension into the C dimension and use a groupNormalizationLayer instead, with the groups defined so that different T belong to different groups.
  7 件のコメント
Matt J
Matt J 2023 年 3 月 15 日
That happens sometimes, but usually you have to submit a formal enhancement request.


Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by