0
$\begingroup$

I have a question regarding the self attention layer of transformers. When dealing with sequences of varying lengths in a mini-batch, we pad sequences so that all sequences in the batch have the same length.

Let's say that that most sequences in a dataset are < 500 elements long, but there are a few very long sequences that can be 1000s of elements long. If I want to handle those very long sequences without truncating, will the multi-head self attention layer's size have to be tailored to the longest possible sequence even when input batches don't contain any of the long sequences?

$\endgroup$

1 Answer 1

1
$\begingroup$

will the multi-head self attention layer's size have to be tailored to the longest possible sequence even when input batches don't contain any of the long sequences?

No.

The attention is automatically tailored to the length of the batch, not the maximum possible length.

Furthermore, with a technique called “bucketing” you create batches with similar lengths to avoid wasting space of the batch with padding tokens.

Deep learning frameworks like Tensorflow and Pytorch make it easy to add bucketing to your data loading logic.

Original answer

will the multi-head self attention layer's size have to be tailored to the longest possible sequence even when input batches don't contain any of the long sequences?

Yes.

However, you normally use "bucketing". This technique consists of creating batches with similar lengths, to avoid wasting space of the batch with padding tokens.

Deep learning frameworks like Tensorflow and Pytorch make it easy to add bucketing to your data loading logic.

$\endgroup$
3
  • $\begingroup$ Follow up question. So if I understand correctly, if I'm able to construct batches with similar lengths, the attention's complexity will only scale with the size of the batch's sequence length? $\endgroup$
    – Murad
    Sep 9, 2022 at 15:27
  • $\begingroup$ The attention complexity scales with the square of the sequence length. The only point in bucketing is not wasting compute with mostly-padding batches. $\endgroup$
    – noe
    Sep 9, 2022 at 16:03
  • $\begingroup$ Ahhh wait wait. I just realized I misunderstood your question. I have edited my answer. $\endgroup$
    – noe
    Sep 9, 2022 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.