Why doesn't concatLayer in Deep Learning Toolbox concatenate the 'T' dimension?

3 ビュー (過去 30 日間)
John Smith
John Smith 2023 年 3 月 13 日
編集済み: Catalytic 2023 年 3 月 23 日
While implementing a ViT transformer in Matlab, I found at that the concatLayer does not concatenate over the T dimension. This is needed to concatenate the class token with patch tokens, since the natural representation is CBT with C corresponding to features, B to batch and T to token within a batch (this is also the canonical representation in the attention function).
It's possible to work around this by hacking to e.g. SCB, but then other problems pop up which also need to be hacked around.

回答 (1 件)

Ben 2023 年 3 月 14 日
You can create a layer that concatenates on the T dimension with functionLayer
sequenceCatLayer = functionLayer(@(x,y) cat(3,x,y));
This will work in dlnetwork to concatenate two CBT dlarray-s.
Since you're concatenating the class token, it might also be worth considering creating a custom layer that has the class token embedding as a Learnable property, and performs the concatenation in the predict method.
  2 件のコメント
Catalytic 2023 年 3 月 23 日
編集済み: Catalytic 2023 年 3 月 23 日
@John Smith - Since Ben's answer yielded a solution for you, you should hit the Accept this Answer button, and likewise with other answers you might not have accepted.


Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by