1
$\begingroup$

I was reading the GPT original paper here and in section 3.5 they mention evaluating on the CoQA dataset. I checked GPT has a sequence length of 512, yet most of the sequences in the CoQA are a few thousand tokens long. So how would they evaluate on this?

$\endgroup$
5
  • $\begingroup$ Hi @AmeenIzhac, welcome to the site. In the GPT-2 paper you referenced, they don't train on CoQA, they just evaluate the model on it in a zero-shot way. About the token length of the data, I have checked a few CoQA entries and all were around 400 tokens using the GPT-3 tokenizer; can you describe how you measured the length you mentioned? $\endgroup$
    – noe
    Apr 14 at 19:01
  • $\begingroup$ Hi @noe, thanks for responding. Yes sorry I meant evaluating. If you look at the dataset here huggingface.co/datasets/stanfordnlp/coqa you will find it gives a bar chart of sample sizes and most are above 512 $\endgroup$ Apr 14 at 20:53
  • $\begingroup$ Those numbers are characters, not tokens, right? $\endgroup$
    – noe
    Apr 14 at 21:52
  • $\begingroup$ @noe, thank you I didn't realize that. In that case almost all samples should be within the models size. But could you perhaps tell me what is normally done when the samples are too long? $\endgroup$ Apr 15 at 10:19
  • $\begingroup$ There is no general answer to that question because it depends on whether you are using the samples to train or to evaluate (like in this case), on the specific task, etc. For instance, if you are training a language model (in a causal language modelling task), then you can simply split the samples, while for QA you normally can't do that. $\endgroup$
    – noe
    Apr 15 at 10:28

1 Answer 1

1
$\begingroup$

In the GPT-2 paper you referenced, they don't train on CoQA, they just evaluate the model on it in a zero-shot way.

About the token length of the data, I have checked a few CoQA entries and all were around 400 tokens using the GPT-3 tokenizer. Also, given the distribution of the sequence length in characters, it seems that all of the samples would probably fit in the context window.

On the matter of how to handle too long samples, there is no general answer, because it depends on whether you are using the samples to train or to evaluate (like in this case), on the specific task, etc. For instance, if you are training a language model (in a causal language modelling task), then you can simply split the samples, while for QA you normally can't do that.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.