I'm trying to train an LSTM for sentiment analysis on the IMDb review dataset.

As input to the word embedding layer, I transform each review to a list of indices (that corresponds to word index in the vocabulary set). I thought of converting the text into one-hot/count matrix, but I will end up with huge sparse matrix (should I worry about this?).

Here is how I am creating the network architecture:

model = Sequential()
model.add(Dense(1, activation='softmax'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy', 'binary_accuracy'])

Here is the model summary:

Layer (type)                 Output Shape              Param #   
embedding_2 (Embedding)      (None, 1422, 4)           201764    
lstm_2 (LSTM)                (None, 1)                 24        
dense_2 (Dense)              (None, 1)                 2         
Total params: 201,790
Trainable params: 201,790
Non-trainable params: 0

Now when I try to train the model I see accuracy stuck at 50%

losses = model.fit(
    x                = term_idx_train,
    y                = y_train,
    epochs           = epochs,
    batch_size       = batch_size,
    validation_split = 0.01

Here is the epochs output:

Epoch 1/10
25000/25000 [==============================] - 1148s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 2/10
25000/25000 [==============================] - 1156s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 3/10
25000/25000 [==============================] - 1149s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 4/10
25000/25000 [==============================] - 1110s 44ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 5/10
16800/25000 [===================>..........] - ETA: 6:10 - loss: 7.9816 - acc: 0.4993 - binary_accuracy: 0.4993

Changing the activation function to a sigmoid and the LSTM blocks to 32 didn't help mush (with 1 epoch):

Train on 24750 samples, validate on 250 samples
Epoch 1/1
24750/24750 [==============================] - 1186s 48ms/step - loss: 0.6932 - acc: 0.5022 - binary_accuracy: 0.5022 - val_loss: 0.6951 - val_acc: 0.0000e+00 - val_binary_accuracy: 0.0000e+00

Epoch 00001: val_loss improved from inf to 0.69513, saving model to sentiment_model

Looking at what the LSTM is predicting, I see:

count   25000.000000
mean    0.499023
std 0.000013
min 0.499010
25% 0.499010
50% 0.499010
75% 0.499010
max 0.499443

Any idea why it's doing this? and how I could fix the issue?

  • $\begingroup$ Can you please explain why did you decide to use 1 unit in the LSTM layer? $\endgroup$ Oct 21, 2018 at 23:32
  • $\begingroup$ trying to output one element that could represent the review do you recommend something else? $\endgroup$
    – bachr
    Oct 22, 2018 at 8:30
  • 1
    $\begingroup$ The output is based on your last dense layer's neuron, and not the LSTM's neuron. I recommend increasing the LSTM neurons to, somewhere around 32, or 16, and then try to compare the results. $\endgroup$ Oct 22, 2018 at 9:46
  • 1
    $\begingroup$ One more thing, even tho your problem is a binary classification, it'll still be a reasonable approach that you one-hot-encode your binary featuers, e-g [1, 0] for positive, [0, 1] for negative, and then have two neurons in the dense layer with softmax activation. As per my experience, that also worked reasonably good. $\endgroup$ Oct 23, 2018 at 1:23
  • 1
    $\begingroup$ Try a very low learning rate, say 1e-6. This ciuld happen by diverging. Also try to reset the weights and reshuffle. May be a local minimum. $\endgroup$
    – Gulzar
    Oct 23, 2018 at 17:37

4 Answers 4


activation = ‘softmax’ should be used for multiclass classification whereas ’sigmoid’ for binary classification.

You can refer to: http://dataaspirant.com/2017/03/07/difference-between-softmax-function-and-sigmoid-function/

If changing the activation function does not help, I will be around for alternative solutions.

  • 1
    $\begingroup$ It didn't help much, still have bad accuracy; I updated the the question above. $\endgroup$
    – bachr
    Oct 22, 2018 at 17:18

Could be a simple error you did in the code (maybe while extracting the dataset) that we can't see in your code sample. The constant loss you showed is a very weird behaviour indeed.


You are trying something really ambitious without a pre-trained embedding like word2vec and an architecture so simple.

I suggest you to give a look at my github repo where (if you are really interested in not using pre-trained embedding) there is an example that starts with random embedding and adjust it while training reaching 87.72% on the TestSet with a CNN. You can easily convert it to LSTM.

  • $\begingroup$ As of the data preprocessing, I'm passing to the embedding layer a list of word indices, I thought with one hot encoding the sentences will end up creating a huge sparse matrix. $\endgroup$
    – bachr
    Oct 23, 2018 at 10:19
  • $\begingroup$ the list of indices is not a very good idea because you are giving an arbitrary ordering to the words! And yes you are right with one-hot-encoding you would end up with a big sparse matrix. But you can try! I still think the best approach would be to use random vectors for the embedding of the words. $\endgroup$ Oct 23, 2018 at 14:25
  • $\begingroup$ I thought that if I don't provide weights to the Embedding it will learn them and they will be initialized randomly, isn't the case? $\endgroup$
    – bachr
    Oct 23, 2018 at 16:46

With Softmax as activation in final layer, you should have n neurons, where n is the number of classes. Here is an explanation: https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/softmax So basically:

model.add(Dense(n_classes, activation='softmax'))

If you are using one hot encoding:

model.add(Dense(y_train[1], activation='softmax'))

It turns out the problem was related to the output_dim of the Embedding layer which was first 4, increasing this to up to 16 helped the accuracy to takeoff to around 96%.

The new problem is the network started overfitting, adding Dropout layers helped reducing this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.