1
$\begingroup$

I am trying to train a LSTM model, but the problem is that the loss and val_loss are decreasing from 12 and 5 to less than 0.01, but the training set acc = 0.024 and validation set acc = 0.0000e+00 and they remain constant during the training. Why this happening and how can I fix it?

$\endgroup$
5
$\begingroup$

I think this is because your targets y are continuous instead of binary. Therefore, either ignore the accuracy report, or binarize your targets if applicable.

I assumed you are using Keras. When you use metrics=['accuracy'], this is what happens under the hood:

if metric in ('accuracy', 'acc'):
   metric_fn = metrics_module.binary_accuracy

where

def binary_accuracy(y_true, y_pred):
    return K.mean(K.equal(y_true, K.round(y_pred)), axis=-1)

In the case of continuous targets, only those y_true that are exactly 0 or exactly 1 will be equal to model prediction K.round(y_pred)). Therefore, accuracy cannot be used for continuous targets.

Here is a code that demonstrates this issue:

from keras import Sequential
from keras.layers import LSTM, Dense
import numpy as np

# Parameters
N = 1000
halfN = int(N/2)
seq_len = 10
dimension = 2
lstm_units = 3

# Data
np.random.seed(123)  # to generate the same numbers

X_zero = np.random.normal(0, 1, size=(halfN, seq_len, dimension))
y_binary_zero = np.zeros((halfN, 1))  # output is only 0
y_continuous_zero = np.random.randint(0, 50, (halfN, 1)) / 100  # output is in [0, 0.5]

X_one = np.random.normal(1, 1, size=(halfN, seq_len, dimension))
y_binary_one = np.ones((halfN, 1))  # output is only 1
y_continuous_one = 0.5 + np.random.randint(0, 50, (halfN, 1)) / 100  # output is in [0.5, 1.0]

p = np.random.permutation(N)  # to shuffle zero and one classes
X = np.concatenate((X_zero, X_one))[p]
y_binary = np.concatenate((y_binary_zero, y_binary_one))[p]
y_continuous = np.concatenate((y_continuous_zero, y_continuous_one))[p]

# Build model
model = Sequential()
model.add(LSTM(lstm_units, input_shape=(None, dimension)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model.summary())

# Fit model
# fit using binary outputs
print('-----------------------Binary---------------------')
model.fit(X, y_binary, batch_size=32, epochs=10)
# fit using continuous outputs
print('-----------------------Continuous---------------------')
model.fit(X, y_continuous, batch_size=32, epochs=10)

which outputs

...
-----------------------Binary---------------------
...
1000/1000 [==============================] - 0s 122us/step - loss: 0.3989 - acc: 0.9500
-----------------------Continuous---------------------
...
1000/1000 [==============================] - 0s 135us/step - loss: 0.5759 - acc: 0.0050
$\endgroup$
  • 1
    $\begingroup$ This feels very likely to be the case. This person had similar results from using MSE as loss but accuracy as a metric: datascience.stackexchange.com/questions/48346/… $\endgroup$ – Simon Larsson Apr 11 at 14:22
  • $\begingroup$ Thank you Esmailian. I must read your code precisely. But just wanted to add, YES. I use LSTM network for stock price prediction and my inputs and labels both are arrays of float numbers like [100.0000 101.2900 99.8956 ....] $\endgroup$ – user145959 Apr 11 at 17:52
1
$\begingroup$

To answer this question I should clarify what cost (loss) function and what evaluation metric's function are.

Cost (loss) function

the cost function is a function that measures the average dissimilarity between your target samples (labels) and the outputs of your network (when it is fed by your feature vectors). Your machine learning algorithm tries to minimize the cost function's value during training process (when your network is fed by training feature vectors only). So in the case of LSTM network, it tries to tweak LSTM weights in each epoch to decrease the cost function's value calculated on training samples. In your problem, depending on the number of your labels, the cost function can be cross-entropy or binary cross-entropy for more than two classes or two classes cases, respectively.

Evaluation metric's function

Evaluation metric's function is a function that measures the average similarity between your target training samples or validation samples and the outputs of your network (when it is fed by your training or validation feature vectors). Unlike the cost function, your machine learning algorithm does not use evaluation metric's function to tweak the LSTM network weights. Instead, it uses evaluation metric's function just to evaluate the model ability to predict the class labels when given feature vectors as input. In your problem, accuracy is the evaluation metric.

So, you should not be surprised if the training_loss and val_loss are decreasing but training_acc and validation_acc remain constant during the training, because your training algorithm does not guarantee that accuracy will increase in every epoch. That's because it does not inspect accuracy to tweak the model's weights, instead it inspect training_loss to do it.

$\endgroup$
  • $\begingroup$ Thanks pythinker but I couldn't understand well enough! $\endgroup$ – user145959 Apr 11 at 17:46
  • $\begingroup$ @pythinker I'm slightly confused about what you said. While I agree with your points about the model using the loss to train the weights, the value of the loss function in turn depends on how much your model gets wrong correct? The more incorrect predictions it makes, the higher the loss and as such the lower the accuracy and vice versa. By that definition, as the loss decreases, the accuracy should increase, or is my understanding incorrect? $\endgroup$ – Binoy Dalal Jul 19 at 4:27
  • $\begingroup$ stackoverflow.com/a/46331458/1822643 explains what i was asking about. you're right in your explanation @pythinker $\endgroup$ – Binoy Dalal Jul 19 at 4:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.