# Keras LSTM accuracy stuck at 50%

I'm trying to train an LSTM for sentiment analysis on the IMDb review dataset.

As input to the word embedding layer, I transform each review to a list of indices (that corresponds to word index in the vocabulary set). I thought of converting the text into one-hot/count matrix, but I will end up with huge sparse matrix (should I worry about this?).

Here is how I am creating the network architecture:

model = Sequential()
input_dim=vocab_size,
output_dim=word_embed_vector_size,
input_length=sentence_len_max)
)
model.summary()


Here is the model summary:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
embedding_2 (Embedding)      (None, 1422, 4)           201764
_________________________________________________________________
lstm_2 (LSTM)                (None, 1)                 24
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 2
=================================================================
Total params: 201,790
Trainable params: 201,790
Non-trainable params: 0
___________________________


Now when I try to train the model I see accuracy stuck at 50%

losses = model.fit(
x                = term_idx_train,
y                = y_train,
epochs           = epochs,
batch_size       = batch_size,
validation_split = 0.01
)


Here is the epochs output:

Epoch 1/10
25000/25000 [==============================] - 1148s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 2/10
25000/25000 [==============================] - 1156s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 3/10
25000/25000 [==============================] - 1149s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 4/10
25000/25000 [==============================] - 1110s 44ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 5/10
16800/25000 [===================>..........] - ETA: 6:10 - loss: 7.9816 - acc: 0.4993 - binary_accuracy: 0.4993


Changing the activation function to a sigmoid and the LSTM blocks to 32 didn't help mush (with 1 epoch):

Train on 24750 samples, validate on 250 samples
Epoch 1/1
24750/24750 [==============================] - 1186s 48ms/step - loss: 0.6932 - acc: 0.5022 - binary_accuracy: 0.5022 - val_loss: 0.6951 - val_acc: 0.0000e+00 - val_binary_accuracy: 0.0000e+00

Epoch 00001: val_loss improved from inf to 0.69513, saving model to sentiment_model


Looking at what the LSTM is predicting, I see:

count   25000.000000
mean    0.499023
std 0.000013
min 0.499010
25% 0.499010
50% 0.499010
75% 0.499010
max 0.499443


Any idea why it's doing this? and how I could fix the issue?

• Can you please explain why did you decide to use 1 unit in the LSTM layer? – Syed Ali Hamza Oct 21 '18 at 23:32
• trying to output one element that could represent the review do you recommend something else? – bachr Oct 22 '18 at 8:30
• The output is based on your last dense layer's neuron, and not the LSTM's neuron. I recommend increasing the LSTM neurons to, somewhere around 32, or 16, and then try to compare the results. – Syed Ali Hamza Oct 22 '18 at 9:46
• One more thing, even tho your problem is a binary classification, it'll still be a reasonable approach that you one-hot-encode your binary featuers, e-g [1, 0] for positive, [0, 1] for negative, and then have two neurons in the dense layer with softmax activation. As per my experience, that also worked reasonably good. – Syed Ali Hamza Oct 23 '18 at 1:23
• Try a very low learning rate, say 1e-6. This ciuld happen by diverging. Also try to reset the weights and reshuffle. May be a local minimum. – Gulzar Oct 23 '18 at 17:37

activation = ‘softmax’ should be used for multiclass classification whereas ’sigmoid’ for binary classification.

If changing the activation function does not help, I will be around for alternative solutions.

• It didn't help much, still have bad accuracy; I updated the the question above. – bachr Oct 22 '18 at 17:18

Could be a simple error you did in the code (maybe while extracting the dataset) that we can't see in your code sample. The constant loss you showed is a very weird behaviour indeed.

Anyway...

You are trying something really ambitious without a pre-trained embedding like word2vec and an architecture so simple.

I suggest you to give a look at my github repo where (if you are really interested in not using pre-trained embedding) there is an example that starts with random embedding and adjust it while training reaching 87.72% on the TestSet with a CNN. You can easily convert it to LSTM.

• As of the data preprocessing, I'm passing to the embedding layer a list of word indices, I thought with one hot encoding the sentences will end up creating a huge sparse matrix. – bachr Oct 23 '18 at 10:19
• the list of indices is not a very good idea because you are giving an arbitrary ordering to the words! And yes you are right with one-hot-encoding you would end up with a big sparse matrix. But you can try! I still think the best approach would be to use random vectors for the embedding of the words. – Francesco Pegoraro Oct 23 '18 at 14:25
• I thought that if I don't provide weights to the Embedding it will learn them and they will be initialized randomly, isn't the case? – bachr Oct 23 '18 at 16:46

With Softmax as activation in final layer, you should have n neurons, where n is the number of classes. Here is an explanation: https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/softmax So basically:

model.add(Dense(n_classes, activation='softmax'))


If you are using one hot encoding:

model.add(Dense(y_train[1], activation='softmax'))


It turns out the problem was related to the output_dim of the Embedding layer which was first 4, increasing this to up to 16 helped the accuracy to takeoff to around 96%.

The new problem is the network started overfitting, adding Dropout layers helped reducing this.