GPT-2 architecture question

Question

I am currently working on a NLP model that compares two comments and determines which one would be more popular. I have already came up with an architecture - it will be based on GPT-2. But now I am struggling understanding what is the general format of an output of it. I inspected this PyTorch implementation of GPT-2 and here is what I understood:

• GPT2Model is the main transformer block, which uses stack of decoders (class Block).
• Block is just one decoder block with attention and convolution layers
• GPT2LMHead is just some number of fully-connected layers. Simple classification head.

What I don't understand so far is:

1. What is presents variable for? I looked inside and it is just list of tensors, but I can't really figure out what are they.
2. If I want to get an embedding of my input sentence, which class I need to use? I thought it is GPT2Model that returns some hidden states, but it returns matrix with dimensions (batch_size, sentence_length + smth, 768). Why is it a matrix and how to get vector then?
3. What is the purpose of set_embedding_weights method? To be honest, I don't even understand what embedding weights really are.
4. If I want to my output be of fixed shape, what placeholders do I need to use in case when an input sentence is smaller than max input size of the GPT-2 model?

Please, can you help me to understand this? I would appreciate any help. Thank you in advance!

Answer 1

1. presents are the model hidden states. This return value is used during inference, to avoid recomputing the previous steps' hidden states at every inference step (see sample.py).
2. Autoregressive language models do not generate sentence representations, just predict the next token. Typically, it's been masked language models (like BERT) the ones that generated a sentence-level representation, usually training it over some sentence-level task (e.g. next sentence prediction in BERT). Therefore, GPT-2 is simply not meant for what you want.
3. set_embedding_weights is a method of the GPT2LMHead to set the fully connected matrix to be equal to the embedding matrix. This way, you reuse the same parameters for the embedding matrix and for the logit projection matrix. This is a frequent approach to regularize the model (i.e. avoid overfitting reducing the total number of parameters). I believe this is the piece of research that introduced such an approach.
4. As I mentioned in point (2), GPT-2 is not meant for this. One option would be to average the output representations for all tokens. Another option would be to use a model that is specifically designed to obtain sentence-level representations, like BERT.