LSTM padding and masking

I am solving a sequence-to-sequence classification problem based on LSTM using Matlab 2020b. The sequences have varaible length so padding within each minibatch is needed. However, I am not sure if Matlab automatically do the masking when calculating the crossentroy loss as well as the training/validation accuracy. From the training plot, the reported accuracy (around 70%) is much lower than those manually calculated by using checkpoints (where I get around 90% accuracy). I suspect although Matlab 2020b supports sequence padding and validation data in LSTM, it still did not offer the option of masking to reduce the influence caused by padding. Any insights?

Answer 1

[Aditya Patil]

Currently, masking is not supported in MATLAB. I have brought the request to the notice of concerned people.

As a workaround, you can sort the inputs so that the amount of padding required is minimized. You may also set the minibatch size to 1, so that no padding is required.

[Yildirim Kocoglu]

Thank you! I was really curious about this as well since it can be done in python. I really hope they can add this feature.