Passing target text to gpt2 and T5 for fine tuning to learn text generation task

Question

I have text with each line in following format:

<text-1> some text-1 <text-2> some text-2 <text-3> some text-3

I want fine tune model to learn generate some text-3 after reading some text-1 and some text-2. In GPT2 and T5 text generation tutorials, we do specify input-ids for target text i.e. labels, but for GPT2 we dont.

For example in this T5 text generation tutorial, we can find line:

model_inputs["labels"] = labels["input_ids"]

But I could not find any such line in these GPT2 text generation examples:

• huggingtweets demo,

• huggingartists demo

• Finetune GPT2 for text generation

Answer 1

Note that:

• In your first and second GPT-2 links, the logic to feed data to the model is handled by the Trainer class, that's why they don't need to explicitly prepare the input and output data and give it to the model.

• In your third GPT-2 link, you can find the place where the expected output (i.e. labels) is passed to the model (which internally shifts them to meet the actual expectations of the Transformer decoder):

  outputs = model(input_tensor, labels=input_tensor)
  

Each implementation is different, even for the same model. Looking for known structures in the code is usually effective, but sometimes it is not, and you need to actually dive into the code to understand what it does.