0
$\begingroup$

I have a basic doubt. Kindly clarify this.

My doubt is, When we are using LSTM's, We pass the words sequentially and get some hidden representations.

Now transformers also does the same thing except non sequentially. But I have seen that the output of BERT based models can be used as word embeddings.

Why can't we use the output of LSTM also as a word embedding? I can find sentence similarity and all with LSTM also ?

For eg : If I have a sentence " is it very hot out there"

Now I will apply word2Vec and get dense representations and pass it to my LSTM model. The output of my LSTM can also be used as word embeddings as we do the same with BERT?

My understanding was that LSTM is used to identify dependencies between words and using that learned weights to perform classification/similar tasks.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

There are multiple concepts mixed in your question.

  • Contextual vs. non-contextual word embeddings: word2vec is a non-contextual approach to obtaining token embeddings. This means that a specific word has the same vector representation regardless of the other words in the sentence it appears. BERT, on the other hand, can be used to obtain contextual representations, because the representations of a token depend directly on the other words in the sentence.

  • Contextual word embeddings with LSTMs. You can obtain contextual word embeddings with LSTMs. Actually, BERT has 2 predecessors that are just that. These models are ULMFit and ELMo. Both are bidirectional LSTMs. The fact that they are bidirectional is important here, otherwise, the representations would only be contextual for the words to the right of each word.

  • Using BERT or LSTMs for classification and other tasks. Both BERT and LSTMs are suitable to perform text classification.

    In the case of BERT, the sentence-level representation is obtained by prefixing the sentence with the special token [CLS] and taking the representations obtained at that position as sentence representation (this is trained with the next-sentence prediction task in the BERT training procedure).

    In the case of LSTMs, the sentence-level representation is usually obtained either as the last output of a unidirectional LSTM or by global pooling over all the representations of a bidirectional LSTM.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Just a small doubt, If I am passing a word2Vec to my Bi Directional LSTM and obtain output for each token/word, Would that be considered a contextual embedding? $\endgroup$
    – mewbie
    Jul 29, 2022 at 10:31
  • $\begingroup$ Yes, those would be contextual representations. $\endgroup$
    – noe
    Jul 29, 2022 at 10:35
  • $\begingroup$ Now it makes real sense. So my LSTM model uses the Word2Vec to obtain some dense representations for each word by adjusting the weights. Once the model is trained, The contextual richness of the words are good enough to classify the sentence without any ambiguity. Right? $\endgroup$
    – mewbie
    Jul 29, 2022 at 10:38
  • $\begingroup$ One last thing, If I am feeding a One hot vector to Bi directional LSTM, Would that output be contextual? $\endgroup$
    – mewbie
    Jul 29, 2022 at 10:39
  • $\begingroup$ Yes, by feeding one-hot vectors to a bidirectional LSTM you would obtain contextual representations $\endgroup$
    – noe
    Jul 29, 2022 at 10:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.