How to Implement padding and masking sequences for RNN

Question

As an exercise, I'm building a network for binary classification of sequences (whether a sequence belongs to type A or type B). The network consists of an RNN with one LSTM layer, and on top of it an MLP that outputs the classification. I input batches of sequences with different lengths into the network, which means I need to pad the sequences to make them equal in length, and to mask the outputs of the network to make them the same length as the original sequences.

What is the correct way to implement padding/masking in PyTorch? I have read about functions like pad_sequence(), pack_sequence(), pack_padded_sequence(), etc., but I have already become confused with all these functions... Or is there any other "secret" way that I don't know of?

Answer 1

Welcome to the forum @kodkod! In the context of an encoder-decoder architecture, I’ve used the following approach. Your problem is slightly different so your mileage may vary. I had a pad_sentences method – so find the largest sentence in each batch, and pad the other sentences to be the same length. I was doing this by manually appending pad tokens before embedding them, but pytorch has a pad_sequence function which will stack a list of tensors and then pad them. Generate_sentence_masks – this takes encodings, the list of actual source lengths, and returns a tensor which contains 1s in the position where there was an actual token, and 0s in positions that were padded. These sentence masks are then passed to the decode method along with the encoder hidden states, decoder initial state, and the padded targets. In the encode method, the padded input is embedded, and then packed with pack_padded_sequence. The packed padded sequence is then run through the encoder LSTM to generate the hidden states. There are a few good tutorials on LSTMs including one here that does sentiment analysis with LSTMs: https://www.kaggle.com/code/arunmohan003/sentiment-analysis-using-lstm-pytorch

hth.