I am trying to build LSTM NN to classify the sentences. I have seen many examples where sentences are converted to word vectors using glove, word2Vec and so on here is an example of it. This solution works, on the similar lines I wrote the below code which uses Universal Sentence encoder to generate the embedding of the entire sentence and use that with LSTM NN to classify the sentences but it is not working even after 200 epochs the model doesn't converge.
Please find the code below
import tensorflow as tf import keras from keras.layers import Input, LSTM, Dense, Activation, Dropout,Embedding from keras.models import Model import pandas as pd import tensorflow_hub as hub encoder=hub.load("https://tfhub.dev/google/universal-sentence-encoder/4") word = "Elephant" sentence = "I am a sentence for which I would like to get its embedding." paragraph = ( "Universal Sentence Encoder embeddings also support short paragraphs. " "There is no hard limit on how long the paragraph is. Roughly, the longer " "the more 'diluted' the embedding will be.") messages = [word, sentence, paragraph,word, sentence, paragraph] labels = [0, 1, 2,0, 1, 2] reviews = tf.one_hot(labels, depth=3) train_embed=encoder(messages) embedding_layer = Embedding(6,512,trainable=False) embedding_layer.build((None,)) embedding_layer.set_weights([train_embed]) input=Input(shape=(512,),dtype="float64") X=embedding_layer(input) X=LSTM(units=128,return_sequences=True)(X) X=Dropout(rate=0.5)(X) X=LSTM(units=128)(X) X=Dense(units=3)(X) X=Activation("softmax")(X) model=Model(inputs=input,outputs=X) model.compile(loss="categorical_crossentropy",optimizer="adam",metrics=["accuracy"]) model.fit(train_embed,reviews,epochs = 200,shuffle=True)
The out put looks like below
Epoch 196/200 1/1 [==============================] - 1s 534ms/step - loss: 1.1557 - accuracy: 0.3333 Epoch 197/200 1/1 [==============================] - 1s 501ms/step - loss: 1.0919 - accuracy: 0.5000 Epoch 198/200 1/1 [==============================] - 1s 518ms/step - loss: 1.2014 - accuracy: 0.0000e+00 Epoch 199/200 1/1 [==============================] - 1s 501ms/step - loss: 1.1008 - accuracy: 0.3333 Epoch 200/200 1/1 [==============================] - 1s 512ms/step - loss: 1.0509 - accuracy: 0.5000
Why is the model not converging? does sentence encoding doesn't work with LSTM?