0
$\begingroup$

From what I searched online, XLNET model is pre-trained with 512 tokens, and https://github.com/zihangdai/xlnet/issues/80 , I didn't find too much useful information on that either. How does XLnet outperform BERT on long text when the max_sequence_length hyperparameter is less than 1024 tokens ?

$\endgroup$
0
1
$\begingroup$

BERT also has the same limit of 512 tokens.

Normally, for longer sequences, you just truncate to 512 tokens.

The limit is derived from the positional embeddings in the Transformer architecture, for which a maximum length needs to be imposed. The magnitude of such a size is related to the amount of memory needed to handle texts: attention layers scale quadratically with the sequence length, which poses a problem with long texts.

There are new architectures that specifically deal with longer sequences, like Longformer, Reformer, Big Bird and Linformer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.