# What's the right input for gpt-2 in NLP

I'm fine-tuning pre-trained gpt-2 for text summarization. The dataset contains 'text' and 'reference summary'. So my question is how to add special tokens to get the right input format. Currently I'm thinking doing like this:

example1 <BOS> text <SEP> reference summary <EOS> ,
example2 <BOS> text <SEP> reference summary <EOS> , .....

Is this correct? If so, a follow-up question would be whether the max-token-length(i.e. 1024 for gpt-2) means also the concatenate length of text and reference summary?

Any comment would be very much appreciated!

• Please, consider upvoting the answer if you found it useful, and marking it as correct if deemed so. Alternatively, please considering describing what the answer is lacking or why you think it is not correct, so that it can be improved.
– noe
Jan 9 '21 at 16:07

You would need, nevertheless, to perform some adaptations. Specifically, the original GPT-2 vocabulary does not have the special tokens you use. Instead, it only has <|endoftext|> to mark the end. This means that if you want to use your special tokens, you would need to add them to the vocabulary and get them trained during fine-tuning. Another option is to simply use <|endoftext|> in the places of your <BOS>, <SEP> and <EOS>.
P.S.: I think that your use of <SEP> comes from the fact that other non-generative models like BERT, use similar special tokens ([SEP], [CLS]) and are specifically designed to receive two concatenated segments as input. However, BERT is not a generative language model, in the sense that it was not trained in an autoregressive manner, but with a masked LM loss: