Doubt in ELMO, BERT, Word2Vec

Question

I read an answer on Quora where a NLP Practioner stated that using ELMO and BERT embeddings as input to LSTM or some RNN will defeat the purpose of ELMo and BERT. I am not sure I agree with the above statement.

Normally we pass words to LSTM to obtain context specific represtations and I am aware of this. But, we pass word2vec instead of one-hot because the contextual representation after LSTM processed it will be better. Similarly common sense states that, if we give ELMO or BERT word embeddings to LSTM, It should output more context rich words than word2vec. Aint I right?

I am aware that once the context is obtained we can fine-tune it straight away for some downstream tasks. But why not use it this way in which we pass the context embeddings of ELMo and BERT to an LSTM ?

Doubt #2 :

I saw a post where the author used ELMo Embeddings with average vectors for each word for logistic regression and tree based models. While this worked for them, In general, It doesn't make sense ? because, In Logistic regression, Each parameter is fixed to an input. Like, Theta1*X1. So if X1 is of different word every time, It should ideally be more confusing to the model to fix that parameter compared to TFIDF where we have a fixed index for each word ?

Answer 1

Doubt #1:

You are correct that using ELMo or BERT embeddings as input to an LSTM could potentially lead to better context representation than using Word2Vec. However, the reason the NLP practitioner might argue against using ELMo or BERT with LSTMs is that these models are already designed to capture context in a more advanced way than LSTMs.

ELMo uses a bidirectional LSTM architecture, allowing it to better capture the context of a word in a sentence. BERT, on the other hand, uses the Transformer architecture, which employs self-attention mechanisms to understand the context in a more sophisticated manner.

When you use ELMo or BERT embeddings as input to an LSTM or another RNN, you may lose some of the contextual information already captured by these models. Instead, it would be more effective to fine-tune the ELMo or BERT models directly for your downstream tasks, as they are specifically designed for this purpose and have been shown to achieve state-of-the-art performance.

Doubt #2:

Using ELMo embeddings with average vectors for logistic regression and tree-based models might not be ideal, but it can still work to some extent. Averaging the embeddings of the words in a sentence can capture some of the semantic information, although it will likely lose a significant amount of contextual information.

In the case of logistic regression, the model will indeed learn to associate a specific weight with each input feature. However, these features no longer correspond to fixed words, as they would with a bag-of-words or TF-IDF representation. Instead, the features represent continuous-valued embeddings, which capture semantic relationships between words. As a result, the logistic regression model will learn to associate the weights with the semantic relationships captured by the ELMo embeddings, rather than individual words.

This approach might not be as effective as using more advanced models like BERT or fine-tuning the ELMo model itself, but it can still provide some improvement over traditional bag-of-words or TF-IDF representations, especially when dealing with tasks that benefit from semantic information.