loss/val_loss are decreasing but accuracies are the same in LSTM!

Question

I am trying to train a LSTM model, but the problem is that the loss and val_loss are decreasing from 12 and 5 to less than 0.01, but the training set acc = 0.024 and validation set acc = 0.0000e+00 and they remain constant during the training. Why this happening and how can I fix it?

Answer 1

I think this is because your targets y are continuous instead of binary. Therefore, either ignore the accuracy report, or binarize your targets if applicable.

I assumed you are using Keras. When you use metrics=['accuracy'], this is what happens under the hood:

if metric in ('accuracy', 'acc'):
   metric_fn = metrics_module.binary_accuracy

where

def binary_accuracy(y_true, y_pred):
    return K.mean(K.equal(y_true, K.round(y_pred)), axis=-1)

In the case of continuous targets, only those y_true that are exactly 0 or exactly 1 will be equal to model prediction K.round(y_pred)). Therefore, accuracy cannot be used for continuous targets.

Here is a code that demonstrates this issue:

from keras import Sequential
from keras.layers import LSTM, Dense
import numpy as np

# Parameters
N = 1000
halfN = int(N/2)
seq_len = 10
dimension = 2
lstm_units = 3

# Data
np.random.seed(123)  # to generate the same numbers

X_zero = np.random.normal(0, 1, size=(halfN, seq_len, dimension))
y_binary_zero = np.zeros((halfN, 1))  # output is only 0
y_continuous_zero = np.random.randint(0, 50, (halfN, 1)) / 100  # output is in [0, 0.5]

X_one = np.random.normal(1, 1, size=(halfN, seq_len, dimension))
y_binary_one = np.ones((halfN, 1))  # output is only 1
y_continuous_one = 0.5 + np.random.randint(0, 50, (halfN, 1)) / 100  # output is in [0.5, 1.0]

p = np.random.permutation(N)  # to shuffle zero and one classes
X = np.concatenate((X_zero, X_one))[p]
y_binary = np.concatenate((y_binary_zero, y_binary_one))[p]
y_continuous = np.concatenate((y_continuous_zero, y_continuous_one))[p]

# Build model
model = Sequential()
model.add(LSTM(lstm_units, input_shape=(None, dimension)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model.summary())

# Fit model
# fit using binary outputs
print('-----------------------Binary---------------------')
model.fit(X, y_binary, batch_size=32, epochs=10)
# fit using continuous outputs
print('-----------------------Continuous---------------------')
model.fit(X, y_continuous, batch_size=32, epochs=10)

which outputs

...
-----------------------Binary---------------------
...
1000/1000 [==============================] - 0s 122us/step - loss: 0.3989 - acc: 0.9500
-----------------------Continuous---------------------
...
1000/1000 [==============================] - 0s 135us/step - loss: 0.5759 - acc: 0.0050

Answer 2

To answer this question I should clarify what cost (loss) function and what evaluation metric's function are.

Cost (loss) function

the cost function is a function that measures the average dissimilarity between your target samples (labels) and the outputs of your network (when it is fed by your feature vectors). Your machine learning algorithm tries to minimize the cost function's value during training process (when your network is fed by training feature vectors only). So in the case of LSTM network, it tries to tweak LSTM weights in each epoch to decrease the cost function's value calculated on training samples. In your problem, depending on the number of your labels, the cost function can be cross-entropy or binary cross-entropy for more than two classes or two classes cases, respectively.

Evaluation metric's function

Evaluation metric's function is a function that measures the average similarity between your target training samples or validation samples and the outputs of your network (when it is fed by your training or validation feature vectors). Unlike the cost function, your machine learning algorithm does not use evaluation metric's function to tweak the LSTM network weights. Instead, it uses evaluation metric's function just to evaluate the model ability to predict the class labels when given feature vectors as input. In your problem, accuracy is the evaluation metric.

So, you should not be surprised if the training_loss and val_loss are decreasing but training_acc and validation_acc remain constant during the training, because your training algorithm does not guarantee that accuracy will increase in every epoch. That's because it does not inspect accuracy to tweak the model's weights, instead it inspect training_loss to do it.