I am currently building a encoder-encoder based model with cosine loss function. The dataset is supervised learning.
Here's a basic encoder,
from tensorflow.keras import layers
class LSTMEncoder(layers.Layer):
def __init__(self,
units, # dimensionality of the output space
input_dim, # vocab of size
output_dim, # embedding dimension
name='encoder',
**kwargs):
super(Encoder, self).__init__(name=name, **kwargs)
self.embedding = layers.Embedding(input_dim=input_dim, output_dim=output_dim)
self.lstm = layers.LSTM(units=units)
def call(self, inputs):
emb = self.embedding(inputs)
return self.lstm(emb)
Both the encoders are very similar, and has a output layer since it's a supervised learning, I added a dense layer at the end for probabilities and loss func is consine loss
Now, My goal is not to predict the classes, rather getting the trained embedding for each text sentence from encoder, but encoder has both embedding layer and lstm layer
so I want to know whether to cosider the output of lstm layer as trained embeddings or should I extract embeddings from embedding layer?