I've noticed in one deep pre-trained textual neural network that there are two embedding layers in the beginning and I don't quite understand why there are two of them. As far as I understand (correct me if I'm wrong, I am a newcomer in NLP) in embedding layer there is a vector of trainable weights that forms set of parameters for every unique word. So, what is the meaning of two such layers in a row? Does the second layer create sub-parameters for every parameter in the original embedding layer?
2
-
2$\begingroup$ By definition, you should not have two embedding layers in a row because the input of an embedding layer is integer indices and the output is real vectors, therefore the output of the first embedding layer would not be compatible with the second layer. Can you add a link to the source code? Otherwise, it is difficult to figure it out. $\endgroup$– noeMar 21, 2022 at 11:59
-
$\begingroup$ @noe here is the code to check architecture in which you can see two embedding layers: from transformers import AutoModelForSequenceClassification model = AutoModelForSequenceClassification.from_pretrained('sberbank-ai/rugpt3medium_based_on_gpt2', num_labels=2) print(model) $\endgroup$– Igor IgorMar 21, 2022 at 19:40
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
If you print the model layers with the code you included in the comments, you obtain this:
GPT2ForSequenceClassification(
(transformer): GPT2Model(
(wte): Embedding(50257, 1024)
(wpe): Embedding(2048, 1024)
...
That's where the misunderstanding comes from. Having one layer printed after the other does not imply that they are connected. They are not.
The first embedding is the normal token embedding, with 50257 token IDs. The second embedding is the positional encoding, with 2048 positions.
The model you are exploring is based on GPT-2, which is a Transformer decoder. In such an architecture, the text is encoded as discrete tokens and the resulting embedded vectors are summed together with some special vectors called positional encoding/embeddings, which encodes the position of the token in the sequence:
So, the answer: there aren't two embedding layers in a row, it is a misunderstanding. Actually, one of the embedding layers encodes the tokens and the other encodes the token positions, and the resulting vectors are added together.
