Keras LSTM accuracy stuck at 50%

Question

I'm trying to train an LSTM for sentiment analysis on the IMDb review dataset.

As input to the word embedding layer, I transform each review to a list of indices (that corresponds to word index in the vocabulary set). I thought of converting the text into one-hot/count matrix, but I will end up with huge sparse matrix (should I worry about this?).

Here is how I am creating the network architecture:

model = Sequential()
model.add(Embedding(
    input_dim=vocab_size,
    output_dim=word_embed_vector_size,
    input_length=sentence_len_max)
         )
model.add(LSTM(units=1))
model.add(Dense(1, activation='softmax'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy', 'binary_accuracy'])
model.summary()

Here is the model summary:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_2 (Embedding)      (None, 1422, 4)           201764    
_________________________________________________________________
lstm_2 (LSTM)                (None, 1)                 24        
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 2         
=================================================================
Total params: 201,790
Trainable params: 201,790
Non-trainable params: 0
___________________________

Now when I try to train the model I see accuracy stuck at 50%

losses = model.fit(
    x                = term_idx_train,
    y                = y_train,
    epochs           = epochs,
    batch_size       = batch_size,
    validation_split = 0.01
)

Here is the epochs output:

Epoch 1/10
25000/25000 [==============================] - 1148s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 2/10
25000/25000 [==============================] - 1156s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 3/10
25000/25000 [==============================] - 1149s 46ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 4/10
25000/25000 [==============================] - 1110s 44ms/step - loss: 7.9712 - acc: 0.5000 - binary_accuracy: 0.5000
Epoch 5/10
16800/25000 [===================>..........] - ETA: 6:10 - loss: 7.9816 - acc: 0.4993 - binary_accuracy: 0.4993

Changing the activation function to a sigmoid and the LSTM blocks to 32 didn't help mush (with 1 epoch):

Train on 24750 samples, validate on 250 samples
Epoch 1/1
24750/24750 [==============================] - 1186s 48ms/step - loss: 0.6932 - acc: 0.5022 - binary_accuracy: 0.5022 - val_loss: 0.6951 - val_acc: 0.0000e+00 - val_binary_accuracy: 0.0000e+00

Epoch 00001: val_loss improved from inf to 0.69513, saving model to sentiment_model

Looking at what the LSTM is predicting, I see:

count   25000.000000
mean    0.499023
std 0.000013
min 0.499010
25% 0.499010
50% 0.499010
75% 0.499010
max 0.499443

Any idea why it's doing this? and how I could fix the issue?

Answer 1

activation = ‘softmax’ should be used for multiclass classification whereas ’sigmoid’ for binary classification.

You can refer to: http://dataaspirant.com/2017/03/07/difference-between-softmax-function-and-sigmoid-function/

If changing the activation function does not help, I will be around for alternative solutions.

Answer 2

Could be a simple error you did in the code (maybe while extracting the dataset) that we can't see in your code sample. The constant loss you showed is a very weird behaviour indeed.

Anyway...

You are trying something really ambitious without a pre-trained embedding like word2vec and an architecture so simple.

I suggest you to give a look at my github repo where (if you are really interested in not using pre-trained embedding) there is an example that starts with random embedding and adjust it while training reaching 87.72% on the TestSet with a CNN. You can easily convert it to LSTM.

Answer 3

With Softmax as activation in final layer, you should have n neurons, where n is the number of classes. Here is an explanation: https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/softmax So basically:

model.add(Dense(n_classes, activation='softmax'))

If you are using one hot encoding:

model.add(Dense(y_train[1], activation='softmax'))

Answer 4

It turns out the problem was related to the output_dim of the Embedding layer which was first 4, increasing this to up to 16 helped the accuracy to takeoff to around 96%.

The new problem is the network started overfitting, adding Dropout layers helped reducing this.