Loss value going down while accuracy remains constant?

Question

While I am training, it seems like my loss is going down, but my accuracy remains constant throughout training. It always seems to go towards 0.0023 no matter how I tweak my network, input data length, etc.

My input data is a sequence of sample levels before a current sample level. So basically predict the current sample level based on the previous sample levels.

Network is Sequential:

Dense(seq_length)
LSTM(seq_length*10)
Dense(1)

Validation set is 0.33 of the total input samples.

Using Tensorflow Keras on Windows.

CSV of loss, accuracy, val loss, etc: https://pastebin.com/GPsmeUmg

Answer 1

Everything is fine, loss is going down, your model is learning. The problem is that you are using accuracy on a regression problem.

Accuracy should only be used for classification. It is a very common mistake to make when starting out. If you want another metric then you could use mean_absolute_error which, as the name suggest, tells you how large error your model makes on average.

The reason accuracy does not work is because it only works with exact matches where you have y_true == y_pred. In regression you are working with continuous values which makes this quite rare and will not have much to do with the performance of your model.

The reason you get any accuracy at all is likely because Keras does y_true == round(y_pred), rounding the model prediction. Otherwise accuracy would almost always be zero since the model will never get the same decimals. This question has a good answer by Esmailian that goes a bit more into details on this.

Answer 2

I am assuming you are trying to do classification task using LSTM and your validation accuracy is not increasing.

1. Your model is learning your training data only & making adjustment in the model weights according to it. This is called overfitting in machine learning.
2. So the question is why overfitting happens one of the reason is the limited amount of data is exposed to the model.

I think this is the reason you model is not learning i.e. the limited amount of data.