I am working on a LSTM network that I get loss amounts around 4.7 e-4 . It seems adding more layers and increasing epochs don't help to decreasing it. I also using a Dropout = 0.2 for each of my layers and implemented all the jobs with Keras library.

I like to know about this loss amount? is this large or is OK> Are there any rule of thumb for loss?

And why I can't decrease my loss amount? Is there any problem here?

  • $\begingroup$ 0.2 is very very small. You are almost vanishing the signal. Try something like .85 $\endgroup$ Commented Feb 27, 2019 at 20:10
  • $\begingroup$ @Media: You mean I must eliminate 85% of my hidden units during each iteration? $\endgroup$ Commented Feb 27, 2019 at 20:51
  • 1
    $\begingroup$ You should keep them. .85 means you keep 85 percent of them. $\endgroup$ Commented Feb 27, 2019 at 20:54
  • $\begingroup$ @Wow! I thought the opposite! $\endgroup$ Commented Feb 27, 2019 at 21:06
  • $\begingroup$ Can you give more details? What are your features like? Are you normalizing? BatchNorm? etc? $\endgroup$
    – kylec123
    Commented Feb 27, 2019 at 21:14

1 Answer 1


Although your loss function is an indication of how well the model is training, usually one uses other more intuitive metrics to assess how good the model is.

If you are looking to a classification problem, your loss function is most probably the cross entropy. In what regards the loss function what matters is to understand its behaviour during training, more than its value.

A loss function that reduces its value during training is an indication that the model is effectively training. The loss function will, at some point, start reducing its value, and that means that the model has arrived to a minimum. One need to understand also the interaction of the loss function in training and validation set, and how to detect things like overfitting. If you are not aware of that, there is plenty of literature about the topic.

To know how good a model is, I would use other metrics that give a better indication and intuition. For example, in a classification problem, pne can look to things like Precision, Recall, Accuracy (if the classes are not very unbalance) or even ROC AUC. If it is a regression problem, maybe you are more interested in Mean or Median Absolute Percentage Error (MdAPE or MAPE).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.