0
$\begingroup$

A Transformer, like Roberta, can generate contextual embeddings using the encoder part, similar to a Bidirectional-LSTM that concatenates hidden states. What are the differences between them ? Are there any advantages of Transformer contextual embeddings over Bidirectional-LSTM? Could someone please explain?

$\endgroup$
3
  • $\begingroup$ Are you referring to the differences between the contextual embeddings of Bidirectional LSTMs and contextual embeddings of Transformers or to the differences to the architectures themselves? $\endgroup$
    – noe
    Feb 18 at 15:17
  • $\begingroup$ @noe, I'm asking about the differences between the contextual embeddings of Bidirectional LSTMs and Transformers. Are the contextual embeddings from Transformers better than those from Bidirectional LSTMs ? $\endgroup$
    – user159173
    Feb 18 at 19:54
  • $\begingroup$ @noe, I have edited my question to provide more details $\endgroup$
    – user159173
    Feb 18 at 20:03

2 Answers 2

0
$\begingroup$

Here are some differences:

  • Computational complexity: LSTMs have linear complexity $O(n)$, because you need to process input tokens one by one, while transformers have constant $O(1)$ complexity because all tokens are processed at the same time.
  • Memory: LSTMs have linear memory consumption because you only need to store the last hidden state, while transformers have quadratic memory consumption.
  • Training task: bidirectional LSTMs are trained on a (causal) language modeling task, while Transformer encoders are usually trained on a masked language modeling task.
  • Quality of the embeddings: in general, transformer-based embeddings are preferred quality-wise.
  • Research activity: LSTM's are a much less active area of research and so are LSTM-based embeddings, while transformers have been all the rage since 2017. This means that there are a lot of publicly available pretrained transformer models for embeddings and close to zero LSTM-based models for embeddings.
$\endgroup$
6
  • $\begingroup$ transformer-based embeddings are preferred quality-wise, but what makes the quality of the embeddings better in the Transformer ? $\endgroup$
    – user159173
    Feb 18 at 20:48
  • $\begingroup$ The performance in downstream tasks is better. Transformers are completely different from LSTMs, so there is not an identifiable "differential factor", it's just that the Transformer architecture gives better results in general. $\endgroup$
    – noe
    Feb 18 at 21:36
  • $\begingroup$ Do you mean transformers, such as Roberta, were pre-trained on a large dataset before we could fine-tune them for downstream tasks? What about bidirectional LSTMs, are they also pre-trained on a large dataset ? $\endgroup$
    – user159173
    Feb 19 at 5:23
  • $\begingroup$ Yes, transformers such as BERT and RoBERTa are pre-trained and you can fine-tune them for downstream tasks. Bidirectional LSTMs like ELMo are also pre-trained. $\endgroup$
    – noe
    Feb 19 at 7:12
  • $\begingroup$ Is there any paper or article supporting the claim that transformer-based contextual embeddings are preferred in terms of quality over bidirectional-LSTM, since both architectures can output contextual embedding ? $\endgroup$
    – user159173
    Mar 11 at 8:35
0
$\begingroup$

transformer tech is evolved version of LSTM/rnn based tech by improving on speed and quality. particularly the tokenization, pre-training method, multi-head attention, longer input size are improved or newer techniques from lstm based models. Tokenization addressed issue of scarceness. multiple attention heads to capture different types of inter-relationships between tokens such as morphological, syntactic, and even deeper semantic. This makes the embedding better in terms of how effectively it represents the semantics of the tokens in the given unique context. training on longer sequences captures very refined differences in meaning than older models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.