Ordering training text data by length

Question

If I have text data where the length of documents greatly varies and I'd like to use it for training where I use batching, there is a great chance that long strings will be mixed with short strings and the average time to process each batch will increase because of padding within the batches. I imagine sorting documents naively by length would create a bias of some sort since long documents and short one would tend to be similar to each other. Are there any methods that have been tried that can help reduce training time in this case without sacrificing model performance?

Answer 1

What you are referring to is called "bucketing". It consists of creating batches of sequences with similar length, to minimize the needed padding.

In tensorflow, you can do it with tf.data.Dataset.bucket_by_sequence_length. Take into account that previously it was in different python packages (tf.data.experimental.bucket_by_sequence_length, tf.contrib.data.bucket_by_sequence_length), so the examples online may containt the outdated name.

To see some usage examples, you can check this jupyter notebook, or other answers in stackoverflow, or this tutorial.