What layer to consider for embeddings when building a encoder-encoder model?

Question

I am currently building a encoder-encoder based model with cosine loss function. The dataset is supervised learning.

Here's a basic encoder,

from tensorflow.keras import layers

class LSTMEncoder(layers.Layer):

    def __init__(self,
                 units,  # dimensionality of the output space
                 input_dim,  # vocab of size
                 output_dim,  # embedding dimension
                 name='encoder',
                 **kwargs):
        super(Encoder, self).__init__(name=name, **kwargs)
        self.embedding = layers.Embedding(input_dim=input_dim, output_dim=output_dim)
        self.lstm = layers.LSTM(units=units)

    def call(self, inputs):
        emb = self.embedding(inputs)
        return self.lstm(emb)

Both the encoders are very similar, and has a output layer since it's a supervised learning, I added a dense layer at the end for probabilities and loss func is consine loss

Now, My goal is not to predict the classes, rather getting the trained embedding for each text sentence from encoder, but encoder has both embedding layer and lstm layer

so I want to know whether to cosider the output of lstm layer as trained embeddings or should I extract embeddings from embedding layer?

Answer 1

You either of them:

• If you use the output of the LSTM, you will have contextual embeddings, like BERT or ELMo. This means that each time you want to use your embeddings to encode some input text, you need to pass the input text to your encoder and take the output of the LSTM.
• If you use the embedding table, you will have non-contextual embeddings, like word2vec. This means that each time you want to use your embeddings to encode some input text, you just need to look up the appropriate vector in the embedding table.