I am currently building a encoder-encoder based model with cosine loss function. The dataset is supervised learning.

Here's a basic encoder,

from tensorflow.keras import layers

class LSTMEncoder(layers.Layer):

    def __init__(self,
                 units,  # dimensionality of the output space
                 input_dim,  # vocab of size
                 output_dim,  # embedding dimension
        super(Encoder, self).__init__(name=name, **kwargs)
        self.embedding = layers.Embedding(input_dim=input_dim, output_dim=output_dim)
        self.lstm = layers.LSTM(units=units)

    def call(self, inputs):
        emb = self.embedding(inputs)
        return self.lstm(emb)

Both the encoders are very similar, and has a output layer since it's a supervised learning, I added a dense layer at the end for probabilities and loss func is consine loss

Now, My goal is not to predict the classes, rather getting the trained embedding for each text sentence from encoder, but encoder has both embedding layer and lstm layer

so I want to know whether to cosider the output of lstm layer as trained embeddings or should I extract embeddings from embedding layer?


1 Answer 1


You either of them:

  • If you use the output of the LSTM, you will have contextual embeddings, like BERT or ELMo. This means that each time you want to use your embeddings to encode some input text, you need to pass the input text to your encoder and take the output of the LSTM.
  • If you use the embedding table, you will have non-contextual embeddings, like word2vec. This means that each time you want to use your embeddings to encode some input text, you just need to look up the appropriate vector in the embedding table.
  • $\begingroup$ thank you for the info $\endgroup$
    – user_12
    Jan 20, 2021 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.