0
$\begingroup$

How do transformer-based architectures, such as Roberta, etc., generate contextual embeddings? The issue is, I haven't found any articles that explain this process.

$\endgroup$

1 Answer 1

0
$\begingroup$

Yes, transformer-based architectures generate contextual token embeddings.

In article To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks, we can find the following description of the feature extraction process:

For both ELMo and BERT, we extract contextual representations of the words from all layers. During adaptation, we learn a linear weighted combination of the layers (Peters et al., 2018) which is used as input to a task-specific model. When extracting features, it is important to expose the internal layers as they typically encode the most transferable representations.

It basically says:

  • Run the model with your input in inference mode.
  • Take the output vectors of the model layers, including the middle ones.
  • In your task classifier, learn a linear combination of the layers you took from the previous model.
$\endgroup$
4
  • $\begingroup$ Thanks, but I'm actually asking about the process of creating contextual embeddings, it's about understanding how the input transforms when go to the embedding layer, positional embedding, and various attention layers, etc. $\endgroup$
    – user159173
    Mar 11 at 14:16
  • $\begingroup$ That's basically the transformer architecture. You can check the original transfomer paper or a popular blog post called "The illustrated BERT" $\endgroup$
    – noe
    Mar 11 at 14:47
  • $\begingroup$ Thanks, so the Roberta architecture just a stack of Transformers' encoders, or have there been any changes to the architecture ? $\endgroup$
    – user159173
    Mar 11 at 18:39
  • $\begingroup$ Yes, RoBERTa is just a Transformer encoder. This is described in section 2.2 of the original RoBERTa paper. $\endgroup$
    – noe
    Mar 11 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.