I'm fine-tuning pre-trained gpt-2 for text summarization. The dataset contains 'text' and 'reference summary'. So my question is how to add special tokens to get the right input format. Currently I'm thinking doing like this:

example1 <BOS> text <SEP> reference summary <EOS> ,
example2 <BOS> text <SEP> reference summary <EOS> , .....

Is this correct? If so, a follow-up question would be whether the max-token-length(i.e. 1024 for gpt-2) means also the concatenate length of text and reference summary?

Any comment would be very much appreciated!


1 Answer 1


Updated answer

After reading @Jessica's answer, I carefully read the original GPT-2 paper and I confirm that the authors do not add special tokens, but simply the text TL;DR: (be careful to include the :, which is not present in the referenced answer) after the text to summarize. Therefore, the format would be:

<|endoftext|> + article + " TL;DR: " + summary + <|endoftext|>

Note that in the original GPT-2 vocabulary there is no <|startoftext|> token, and neither there is in the Huggingface implementation.

Old answer

GPT-2 is a causal language model. This means that, by default, it receives either no input at all or the initial tokens of a sentence/paragraph. It then completes whatever it was passed as input. Therefore, it is not meant to be used the way you are trying to do it.

Normally, in order to do conditional text generation, people use an encoder-decoder architecture, that is, a full encoder-decoder Transformer instead of GPT-2, which only has the decoder part.

Nevertheless, while it was not meant to work the way you are using it, it is possible that this works. This kind of thing has been done before, for instance, in this NeurIPS 2018 article that uses only a Transformer decoder for machine translation, concatenating source and target sides, like you do:

enter image description here

You would need, nevertheless, to perform some adaptations. Specifically, the original GPT-2 vocabulary does not have the special tokens you use. Instead, it only has <|endoftext|> to mark the end. This means that if you want to use your special tokens, you would need to add them to the vocabulary and get them trained during fine-tuning. Another option is to simply use <|endoftext|> in the places of your <BOS>, <SEP> and <EOS>.

For GPT-2, there is only a single sequence, not 2. Therefore, the maximum token length would apply for the concatenation of text and reference summary.

P.S.: I think that your use of <SEP> comes from the fact that other non-generative models like BERT, use similar special tokens ([SEP], [CLS]) and are specifically designed to receive two concatenated segments as input. However, BERT is not a generative language model, in the sense that it was not trained in an autoregressive manner, but with a masked LM loss:

enter image description here

  • 1
    $\begingroup$ Thank you for the explication, it is so useful. However, in the paper of GPT2 the authors used the model also for automatic summarization task. They explained, it is possible to summarize text adding TL;DR after the article. It means, you will have something like: `` <|startoftext|> + article + 'TL;DR' + summary + <|endoftext|> `` I hope it helps. $\endgroup$ Commented May 24, 2022 at 8:58
  • $\begingroup$ You are totally right, I will edit my answer. $\endgroup$
    – noe
    Commented May 24, 2022 at 10:50
  • $\begingroup$ In my answer I made some amendments to the information you provide. Specifically, the TLDR marker includes a final colon (TL;DR:) and that the <|startoftext|> token is not present in the GPT-2 vocabulary. $\endgroup$
    – noe
    Commented May 24, 2022 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.