0
$\begingroup$

If I have text data where the length of documents greatly varies and I'd like to use it for training where I use batching, there is a great chance that long strings will be mixed with short strings and the average time to process each batch will increase because of padding within the batches. I imagine sorting documents naively by length would create a bias of some sort since long documents and short one would tend to be similar to each other. Are there any methods that have been tried that can help reduce training time in this case without sacrificing model performance?

$\endgroup$

1 Answer 1

1
$\begingroup$

What you are referring to is called "bucketing". It consists of creating batches of sequences with similar length, to minimize the needed padding.

In tensorflow, you can do it with tf.data.Dataset.bucket_by_sequence_length. Take into account that previously it was in different python packages (tf.data.experimental.bucket_by_sequence_length, tf.contrib.data.bucket_by_sequence_length), so the examples online may containt the outdated name.

To see some usage examples, you can check this jupyter notebook, or other answers in stackoverflow, or this tutorial.

$\endgroup$
1
  • 1
    $\begingroup$ That's what I was looking for, thank you! I'm not using TensorFlow, and unfortunately neither PyTorch or huggingface provide a way to bucket like that. The easiest way I found to do that instead was with huggingface datasets, I was able to sort all the texts by length, and then randomly sample the batches using the shard function (here) instead. $\endgroup$
    – Badr Jaidi
    Nov 13, 2022 at 1:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.