Sentence embeddings with LSTM to classify the sentences is not working

Question

I am trying to build LSTM NN to classify the sentences. I have seen many examples where sentences are converted to word vectors using glove, word2Vec and so on here is an example of it. This solution works, on the similar lines I wrote the below code which uses Universal Sentence encoder to generate the embedding of the entire sentence and use that with LSTM NN to classify the sentences but it is not working even after 200 epochs the model doesn't converge.

Please find the code below

import tensorflow as tf
import keras
from keras.layers import Input, LSTM, Dense, Activation, Dropout,Embedding
from keras.models import Model
import pandas as pd
import tensorflow_hub as hub

encoder=hub.load("https://tfhub.dev/google/universal-sentence-encoder/4")

word = "Elephant"
sentence = "I am a sentence for which I would like to get its embedding."
paragraph = (
    "Universal Sentence Encoder embeddings also support short paragraphs. "
    "There is no hard limit on how long the paragraph is. Roughly, the longer "
    "the more 'diluted' the embedding will be.")
messages = [word, sentence, paragraph,word, sentence, paragraph]
labels = [0, 1, 2,0, 1, 2]
reviews = tf.one_hot(labels, depth=3)
train_embed=encoder(messages)


embedding_layer = Embedding(6,512,trainable=False)
embedding_layer.build((None,))
embedding_layer.set_weights([train_embed])

input=Input(shape=(512,),dtype="float64")
X=embedding_layer(input)
X=LSTM(units=128,return_sequences=True)(X)
X=Dropout(rate=0.5)(X)
X=LSTM(units=128)(X)
X=Dense(units=3)(X)
X=Activation("softmax")(X)
model=Model(inputs=input,outputs=X)
model.compile(loss="categorical_crossentropy",optimizer="adam",metrics=["accuracy"])
model.fit(train_embed,reviews,epochs = 200,shuffle=True)

The out put looks like below

Epoch 196/200
1/1 [==============================] - 1s 534ms/step - loss: 1.1557 - accuracy: 0.3333
Epoch 197/200
1/1 [==============================] - 1s 501ms/step - loss: 1.0919 - accuracy: 0.5000
Epoch 198/200
1/1 [==============================] - 1s 518ms/step - loss: 1.2014 - accuracy: 0.0000e+00
Epoch 199/200
1/1 [==============================] - 1s 501ms/step - loss: 1.1008 - accuracy: 0.3333
Epoch 200/200
1/1 [==============================] - 1s 512ms/step - loss: 1.0509 - accuracy: 0.5000

Why is the model not converging? does sentence encoding doesn't work with LSTM?

Answer 1

The problem is that you are encoding the pieces of text as vectors and feeding those vectors to the model, but then the first layer of the model is again an embedding layer.

You should only use the embedding once: either you embed the text out of the model (train_embed=encoder(messages)) or you pass integer inputs to the model and them inside the model (X=embedding_layer(input)), but not both.

The problem is masked by the fact that the embedding layer does not raise an error when it receives an input of type float. Instead, it silently casts the input to integer.

Apart from the problem itself, there are some extra things that you should note:

• depending on what you do, you may need to adjust the input to the LSTM to make it have the 3 dimensions it expects.
• Here, the universal sentence encoder creates a fixed size vector for each sequence, and then you are feeding such fixed-size vector to an LSTM. It doesn't make much sense to have LSTMs process non-sequential data. I think you could just apply an MLP (i.e. a couple of dense layers with ReLU activations plus a final dense with the softmax activation) to the vector.
• You are applying too much dropout for this case where there is almost no data.