5
$\begingroup$

While model tuning using cross validation and grid search I was plotting the graph of different learning rate against log loss and accuracy separately.

Log loss

When I used log loss as score in grid search to identify the best learning rate out of the given range I got the result as follows:

Best: -0.474619 using learning rate: 0.01

  • -0.674328 (0.000482) with: learning rate: 0.0001
  • -0.583335 (0.003236) with: learning rate: 0.001
  • -0.474619 (0.004336) with: learning rate: 0.01
  • -0.494540 (0.008705) with: learning rate: 0.1

Accuracy

When I used accuracy as score in grid search to identify the best learning rate out of the given range I got the result as follows:

Best: 0.781958 using learning rate: 0.1

  • 0.656220 (0.085705) with: learning rate: 0.0001
  • 0.715279 (0.010021) with: learning rate: 0.001
  • 0.740141 (0.007927) with: learning rate: 0.01
  • 0.781958 (0.003770) with: learning rate: 0.1

In both cases I got different learning rates that I should use to tune my model. When the score is log loss, I got optimum setting for learning rate as 0.01. When score is accuracy, I got optimum setting for learning rate as 0.1.

In such cases, what score should I use for my model?

$\endgroup$
4
$\begingroup$

According to me, it is not correct to co-relate loss with accuracy.

Loss is used to optimize the hypothesis such that we can get best weights whereas accuracy is used to identify how well model is doing in term of correctly predicting the values.

Model internally takes the reference of predict_proba() and returns 1 if probability is > .5 otherwise 0. For example if returned predict_proba() is (.49, .51), model will return 1 as an classification output.

Now consider a use case where some trained model is used for test-data prediction. Assumed such model has 100% accuracy but predict_proba() value close to (.49,.51) or (.51,.49) i.e. having very low confidence level.

In such case logloss is quite high, even though it's having 100% accuracy.

If our criteria for model selection is "accuracy value" then we are selecting the bad model.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ That is quite good explanation. Thank you for clearing my confusion. I think logloss tell us how confidently we are predicting the records in any of the given class rather than just calculating correct or incorrect prediction based on accuracy. $\endgroup$ – CodeMaster GoGo Sep 11 '19 at 10:53
  • $\begingroup$ predict_proba will give you confidence score. logloss will be used to accumulate the training set loss and our target is to minimize this loss. Will suggest to read wiki.fast.ai/index.php/Log_Loss...Still have any question, please let me know. $\endgroup$ – vipin bansal Sep 11 '19 at 13:24
1
$\begingroup$

Actually, loss is used by the model to decide the probability of the class. So, logloss just indicates how much is your model certain in comparison to the correct labels of the classes in test samples.

Accuracy indicates what percentage of test samples are classified correctly.

Look at this: What is the relationship between the accuracy and the loss in deep learning?

|improve this answer|||||
$\endgroup$
0
$\begingroup$

It is possible for the loss to drop a bit but the accuracy not to improve at all, due to the abrupt difference in the classification levels (and the fact that the loss is given by a mean).

As with which metric to use when evaluating the model, I think usually you want it to be accuracy (for classification), since it's what matters in the end. We mostly use the logloss to check if everything is ok (i.e. convergence is smooth and monotonically decreasing).*

I believe you probably also used the same number of epochs for the different learning rates, right? In that case, it's a bit natural for the lower learning rate to go a little bit slower. Unless it finds a nasty plateau, it should converge to a lower minimum.

Anyway, when it comes to learning rates, the best practice is to make it smaller as you go through the training process, i.e., learning rate decay. Many practitioners also use cosines to diminish the value of the learning rate and then reset it so that we "never" get stuck in local minima. You can check out the first lecture of the fast.ai course to take a look into another better technique.

*EDIT:

So... My reasoning was a bit oversimplified — which is terrible. Let's go over it again in a little bit more detail. What should you analyse here? The answer is: it depends on your problem. Accuracy is of good guidance, however, if, for example, 90% of your dataset consists of one class, your algorithm might "learn" to give everything that class and call it a day, which will give you 90% accuracy but doesn't mean anything. In order to more easily identify these imbalanced cases, many libraries offer methods for you to compare your classifier to the so-called base-class zero classifier, which can be the basic standard of comparison with respect to performance.

In almost all cases, the recommendation for assessing performance of classifiers is to use the confusion matrix and look for the statistics you most care for. For instance, if you're looking at something like identifying a person has cancer, you want the false positive rate (FPR) to be as low as possible, because misdiagnosing an ill patient can be a disaster and, if someone is not ill, it isn't that a big a problem for him or her to do more tests and be sure about it.

Another tool that is very standard in analysing the performance of classifiers is the ROC-AUC criterion, which creates a graph and index that account for both false positive rate (FPR) and true positive rate (TPR).

Let me reiterate that this is not at all an easy problem and careful analysis needs to be done. Use all the necessary and available tools.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ My dataset is not that large and so i am doing in one go. How can we change the value of learning rate as we approah to minima? I have seen few articles and links where they have explained the concept of gradient descent using big steps and so we reach to minima it takes small step. I think steps is nothing but learning rate then how can we change the learning rate in python during the training? $\endgroup$ – CodeMaster GoGo Oct 18 '18 at 8:16
  • 1
    $\begingroup$ I must admit I haven't really practiced it much so far. Different frameworks will have different approaches to it (TensorFlow uses ` tf.train.exponential_decay` apparently). I don't really like the Keras's one, but they do it like this: $learning\_rate/(1 + decay\_rate \cdot num\_batch)$. You can read more about it here $\endgroup$ – Philippe Fanaro Oct 18 '18 at 13:28
  • 1
    $\begingroup$ I disagree with your thinking that the accuracy is the better metric for a classification task. Accuracy is deceitful at most of the times, since there will be class unbalance problems. Log_loss is scary since you do not know the interpretation from directly looking at it, but test loss decreases it is certain that your model is learning the labels. What I suggest is to use accuracy and log_loss together, (LightGBM and keras allows it to use a combination of both, also for early stopping), if not possible, use log_loss and monitor the accuracy. $\endgroup$ – Ugur MULUK Dec 17 '18 at 8:51
  • 1
    $\begingroup$ I have to agree with Ugur here. Log loss should be preferred in every single case if your goal is to obtain the most discriminating classifier. Lets use accuracy with a 50% threshold for instance on a binary classification problem. Suppose I have two competing classifiers for a dataset with ground truth labels 1,1,0,1. My first classifier gives probability estimates as (0.51, 0.51, 0.48, 0.51) whereas my second gives probability estimates (0.8, 0.9, 0.2, 0.8). Both models have perfect accuracy scores but one clearly is more confident in its decisions. $\endgroup$ – aranglol May 18 '19 at 17:56
  • $\begingroup$ I admit that my answer was oversimplified. For that, I apologize. I hope the edit is helpful and much more accurate. $\endgroup$ – Philippe Fanaro May 18 '19 at 19:32
0
$\begingroup$

If you use backtracking gradient descent, like in my answer in this link

Does gradient descent always converge to an optimum?

, you can find learning rates automatically, without manual fine-tuning.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.