0
$\begingroup$

I am trying to build LSTM NN to classify the sentences. I have seen many examples where sentences are converted to word vectors using glove, word2Vec and so on here is an example of it. This solution works, on the similar lines I wrote the below code which uses Universal Sentence encoder to generate the embedding of the entire sentence and use that with LSTM NN to classify the sentences but it is not working even after 200 epochs the model doesn't converge.

Please find the code below

import tensorflow as tf
import keras
from keras.layers import Input, LSTM, Dense, Activation, Dropout,Embedding
from keras.models import Model
import pandas as pd
import tensorflow_hub as hub

encoder=hub.load("https://tfhub.dev/google/universal-sentence-encoder/4")

word = "Elephant"
sentence = "I am a sentence for which I would like to get its embedding."
paragraph = (
    "Universal Sentence Encoder embeddings also support short paragraphs. "
    "There is no hard limit on how long the paragraph is. Roughly, the longer "
    "the more 'diluted' the embedding will be.")
messages = [word, sentence, paragraph,word, sentence, paragraph]
labels = [0, 1, 2,0, 1, 2]
reviews = tf.one_hot(labels, depth=3)
train_embed=encoder(messages)


embedding_layer = Embedding(6,512,trainable=False)
embedding_layer.build((None,))
embedding_layer.set_weights([train_embed])

input=Input(shape=(512,),dtype="float64")
X=embedding_layer(input)
X=LSTM(units=128,return_sequences=True)(X)
X=Dropout(rate=0.5)(X)
X=LSTM(units=128)(X)
X=Dense(units=3)(X)
X=Activation("softmax")(X)
model=Model(inputs=input,outputs=X)
model.compile(loss="categorical_crossentropy",optimizer="adam",metrics=["accuracy"])
model.fit(train_embed,reviews,epochs = 200,shuffle=True)

The out put looks like below

Epoch 196/200
1/1 [==============================] - 1s 534ms/step - loss: 1.1557 - accuracy: 0.3333
Epoch 197/200
1/1 [==============================] - 1s 501ms/step - loss: 1.0919 - accuracy: 0.5000
Epoch 198/200
1/1 [==============================] - 1s 518ms/step - loss: 1.2014 - accuracy: 0.0000e+00
Epoch 199/200
1/1 [==============================] - 1s 501ms/step - loss: 1.1008 - accuracy: 0.3333
Epoch 200/200
1/1 [==============================] - 1s 512ms/step - loss: 1.0509 - accuracy: 0.5000

Why is the model not converging? does sentence encoding doesn't work with LSTM?

$\endgroup$
1
  • $\begingroup$ Find a bigger dataset that is prelabeled, like a spam set or maybe the imdb movie set. You may have to tweak your learning rate for such a tiny amount of data. $\endgroup$ Mar 16 at 17:30
1
$\begingroup$

The problem is that you are encoding the pieces of text as vectors and feeding those vectors to the model, but then the first layer of the model is again an embedding layer.

You should only use the embedding once: either you embed the text out of the model (train_embed=encoder(messages)) or you pass integer inputs to the model and them inside the model (X=embedding_layer(input)), but not both.

The problem is masked by the fact that the embedding layer does not raise an error when it receives an input of type float. Instead, it silently casts the input to integer.

Apart from the problem itself, there are some extra things that you should note:

  • depending on what you do, you may need to adjust the input to the LSTM to make it have the 3 dimensions it expects.
  • Here, the universal sentence encoder creates a fixed size vector for each sequence, and then you are feeding such fixed-size vector to an LSTM. It doesn't make much sense to have LSTMs process non-sequential data. I think you could just apply an MLP (i.e. a couple of dense layers with ReLU activations plus a final dense with the softmax activation) to the vector.
  • You are applying too much dropout for this case where there is almost no data.
$\endgroup$
2
  • $\begingroup$ I resolved this issue by passing the embeddings directly into LSTM with required 3 dimensions (m,1,512) where m is the number of examples, 1 one time sequence of max length 512. This worked for me $\endgroup$
    – Raj
    Mar 18 at 16:55
  • $\begingroup$ @Raj how did you do that? could you please share the working code $\endgroup$
    – sid8491
    Jul 21 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.