I've noticed in one deep pre-trained textual neural network that there are two embedding layers in the beginning and I don't quite understand why there are two of them. As far as I understand (correct me if I'm wrong, I am a newcomer in NLP) in embedding layer there is a vector of trainable weights that forms set of parameters for every unique word. So, what is the meaning of two such layers in a row? Does the second layer create sub-parameters for every parameter in the original embedding layer?
If you print the model layers with the code you included in the comments, you obtain this:
GPT2ForSequenceClassification( (transformer): GPT2Model( (wte): Embedding(50257, 1024) (wpe): Embedding(2048, 1024) ...
That's where the misunderstanding comes from. Having one layer printed after the other does not imply that they are connected. They are not.
The first embedding is the normal token embedding, with 50257 token IDs. The second embedding is the positional encoding, with 2048 positions.
The model you are exploring is based on GPT-2, which is a Transformer decoder. In such an architecture, the text is encoded as discrete tokens and the resulting embedded vectors are summed together with some special vectors called positional encoding/embeddings, which encodes the position of the token in the sequence:
So, the answer: there aren't two embedding layers in a row, it is a misunderstanding. Actually, one of the embedding layers encodes the tokens and the other encodes the token positions, and the resulting vectors are added together.