Padding sequences for neural sequence models (RNNs)

Question

I am padding sequences for a GRU based classifier that I am building in Keras. I'm wondering if there's any accepted best practice for padding the leading or trailing side of the sequence.

E.g.

sequence = [1,2,3,4]
leading_pad = [0,0,0,1,2,3,4]
trailing_pad = [1,2,3,4,0,0,0]

In other projects I have generally padded the leading end of the sequence as a convention imprinted on me by various blogs, course work and other examples.

My question: is there any research that shows whether or not it matters which side of the sequence you pad? Or am I completely over thinking this?

Any advice would be much appreciated.

Answer 1

Padded values are noise when they are regarded as actual values. For example, a padded temperature sequence [20, 21, 23, 0, 0] is the same as a noisy sequence where sensor has failed to report the correct temperature for the last two readings. Therefore, padded values better be cleaned (ignored) if possible.

Best practice is to use a Mask layer before other layers such as LSTM, RNN,.. to mask (ignore) the padded values. This way, it does not matter if we place them first or last.

Check out this post (my answer) that shows how to pad and mask the sequences with different length (with a sample code). You can experiment with the code to see the effect of removing the mask (treating padded values as actual values) on the deterioration of model performance.

This is the python snippet for quick reference:

model.add(Masking(mask_value=special_value, input_shape=(max_seq_len, dimension)))
model.add(LSTM(lstm_units))

where special_value is the padded value that should not have overlap with actual values.