I am currently working on a NLP model that compares two comments and determines which one would be more popular. I have already came up with an architecture - it will be based on GPT-2. But now I am struggling understanding what is the general format of an output of it. I inspected this PyTorch implementation of GPT-2 and here is what I understood:
- GPT2Model is the main transformer block, which uses stack of decoders (class Block).
- Block is just one decoder block with attention and convolution layers
- GPT2LMHead is just some number of fully-connected layers. Simple classification head.
What I don't understand so far is:
- What is
presents
variable for? I looked inside and it is just list of tensors, but I can't really figure out what are they. - If I want to get an embedding of my input sentence, which class I need to use? I thought it is GPT2Model that returns some hidden states, but it returns matrix with dimensions (batch_size, sentence_length + smth, 768). Why is it a matrix and how to get vector then?
- What is the purpose of set_embedding_weights method? To be honest, I don't even understand what embedding weights really are.
- If I want to my output be of fixed shape, what placeholders do I need to use in case when an input sentence is smaller than max input size of the GPT-2 model?
Please, can you help me to understand this? I would appreciate any help. Thank you in advance!