Predicting a next word from a sentence of a different lenght than seen in training

Question

I am building a custom Decoder-only transformer model, which is being trained on the task of Next Word Prediction. The training procedure is analogous to that of chat GPT models - the input to the model is a sentence of length K (say K=30) and the target is this sentence shifted one to the right, e.g.:

"I would like a cup of" - input

"would like a cup of tea" - output

If I train my model on sentences of a specified lenght, say K=30, how will it perform in inference mode when it is provided much shorter sentences, say of length 3?

Answer 1

This can only be answered with actual experiments with your data, model, and training setup.

However, previous research (see Sequence Length is a Domain: Length-based Overfitting in Transformer Models, published at EMNLP'21) suggests that Transformers do not generalize well to unseen sequence lengths:

We showed in our targeted experiment that vanilla Transformer sequence-to-sequence models have a strong tendency to overfit with regard to the target side length of the training sequences. On a simple algorithmic task, we documented that Transformer can generalize very well to unseen examples within the same length bucket but falls short if the same task is required for input of a different length, shorter or longer. The algorithm of the task, even if very simple, is not learned.