From what I searched online, XLNET model is pre-trained with 512 tokens, and https://github.com/zihangdai/xlnet/issues/80 , I didn't find too much useful information on that either. How does XLnet outperform BERT on long text when the max_sequence_length hyperparameter is less than 1024 tokens ?
BERT also has the same limit of 512 tokens.
Normally, for longer sequences, you just truncate to 512 tokens.
The limit is derived from the positional embeddings in the Transformer architecture, for which a maximum length needs to be imposed. The magnitude of such a size is related to the amount of memory needed to handle texts: attention layers scale quadratically with the sequence length, which poses a problem with long texts.