0
$\begingroup$

I'm reading an article about the Embedding Layer:

The Embedding Layer learns word embeddings from raw text. It is initialized with small random numbers and can be learned simultaneously with a neural network in a supervised way using backpropagation for a specific task, such as text classification.

So, the Embedding Layer learn word embeddings from scratch, correct? In comparison to Roberta's word embeddings, using a pretrained model from HuggingFace's Transformer, are there any differences between them ?

$\endgroup$

1 Answer 1

0
$\begingroup$

An embedding layer is just a building block to be used as part of neural architectures. It is just a lookup table whose purpose is to represent tokens as vectors, and to learn these vectors as part of the training of the model in certain task (e.g. masked language modeling). The input of the embedding layer are token indices, and its output are the vectors associated to those tokens.

Actually, an embedding layer is the first layer of most NLP models, including RoBERTa. What we call "RoBERTa" embeddings are not the internal embedding layer inside the model, but the hidden states of the self-attention layer of the model.

$\endgroup$
3
  • $\begingroup$ So Embedding Layer doesn't output word embeddings ? $\endgroup$
    – user159173
    Feb 19 at 22:53
  • $\begingroup$ An embedding layer does output non-contextual token embeddings. Nevertheless, comparing an embedding layer with RoBERTa is like comparing a wheel with a specific model of car and asking if the wheel moves: the wheel does move, but only as a part of something bigger than itself, like a car. $\endgroup$
    – noe
    Feb 20 at 6:44
  • $\begingroup$ Thanks, I was a confused because the article mentioned that you can generate word embeddings using an Embedding Layer. I suppose the Embedding Layer isn't as effective as Roberta Embeddings. $\endgroup$
    – user159173
    Feb 20 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.