Log loss vs accuracy for deciding between different learning rates?

Question

While model tuning using cross validation and grid search I was plotting the graph of different learning rate against log loss and accuracy separately.

Log loss

When I used log loss as score in grid search to identify the best learning rate out of the given range I got the result as follows:

Best: -0.474619 using learning rate: 0.01

• -0.674328 (0.000482) with: learning rate: 0.0001
• -0.583335 (0.003236) with: learning rate: 0.001
• -0.474619 (0.004336) with: learning rate: 0.01
• -0.494540 (0.008705) with: learning rate: 0.1

Accuracy

When I used accuracy as score in grid search to identify the best learning rate out of the given range I got the result as follows:

Best: 0.781958 using learning rate: 0.1

• 0.656220 (0.085705) with: learning rate: 0.0001
• 0.715279 (0.010021) with: learning rate: 0.001
• 0.740141 (0.007927) with: learning rate: 0.01
• 0.781958 (0.003770) with: learning rate: 0.1

In both cases I got different learning rates that I should use to tune my model. When the score is log loss, I got optimum setting for learning rate as 0.01. When score is accuracy, I got optimum setting for learning rate as 0.1.

In such cases, what score should I use for my model?

Answer 1

According to me, it is not correct to co-relate loss with accuracy.

Loss is used to optimize the hypothesis such that we can get best weights whereas accuracy is used to identify how well model is doing in term of correctly predicting the values.

Model internally takes the reference of predict_proba() and returns 1 if probability is > .5 otherwise 0. For example if returned predict_proba() is (.49, .51), model will return 1 as an classification output.

Now consider a use case where some trained model is used for test-data prediction. Assumed such model has 100% accuracy but predict_proba() value close to (.49,.51) or (.51,.49) i.e. having very low confidence level.

In such case logloss is quite high, even though it's having 100% accuracy.

If our criteria for model selection is "accuracy value" then we are selecting the bad model.

Answer 2

Actually, loss is used by the model to decide the probability of the class. So, logloss just indicates how much is your model certain in comparison to the correct labels of the classes in test samples.

Accuracy indicates what percentage of test samples are classified correctly.

Look at this: What is the relationship between the accuracy and the loss in deep learning?

Answer 3

It is possible for the loss to drop a bit but the accuracy not to improve at all, due to the abrupt difference in the classification levels (and the fact that the loss is given by a mean).

As with which metric to use when evaluating the model, I think usually you want it to be accuracy (for classification), since it's what matters in the end. We mostly use the logloss to check if everything is ok (i.e. convergence is smooth and monotonically decreasing).*

I believe you probably also used the same number of epochs for the different learning rates, right? In that case, it's a bit natural for the lower learning rate to go a little bit slower. Unless it finds a nasty plateau, it should converge to a lower minimum.

Anyway, when it comes to learning rates, the best practice is to make it smaller as you go through the training process, i.e., learning rate decay. Many practitioners also use cosines to diminish the value of the learning rate and then reset it so that we "never" get stuck in local minima. You can check out the first lecture of the fast.ai course to take a look into another better technique.

*EDIT:

So... My reasoning was a bit oversimplified — which is terrible. Let's go over it again in a little bit more detail. What should you analyse here? The answer is: it depends on your problem. Accuracy is of good guidance, however, if, for example, 90% of your dataset consists of one class, your algorithm might "learn" to give everything that class and call it a day, which will give you 90% accuracy but doesn't mean anything. In order to more easily identify these imbalanced cases, many libraries offer methods for you to compare your classifier to the so-called base-class zero classifier, which can be the basic standard of comparison with respect to performance.

In almost all cases, the recommendation for assessing performance of classifiers is to use the confusion matrix and look for the statistics you most care for. For instance, if you're looking at something like identifying a person has cancer, you want the false positive rate (FPR) to be as low as possible, because misdiagnosing an ill patient can be a disaster and, if someone is not ill, it isn't that a big a problem for him or her to do more tests and be sure about it.

Another tool that is very standard in analysing the performance of classifiers is the ROC-AUC criterion, which creates a graph and index that account for both false positive rate (FPR) and true positive rate (TPR).

Let me reiterate that this is not at all an easy problem and careful analysis needs to be done. Use all the necessary and available tools.

Answer 4

If you use backtracking gradient descent, like in my answer in this link

Does gradient descent always converge to an optimum?

, you can find learning rates automatically, without manual fine-tuning.