0
$\begingroup$

I am trying to select a model from different CNN trained models based on some parameter. Initially, I was considering using model confidence score to decide which model is better. Now I am considering using accuracy for the selection of a good model among other models. I need to ask is there a relation between accuracy and confidence score, does it means high accuracy leads to high confidence score and vice versa. Thanks

$\endgroup$
4
  • $\begingroup$ What exactly do you call confidence score ? And why are you using it as the metric to decide which model is the best. It is much more common to use accuracy metric to decide which model is the best. If you want to go further and accuracy metric is not good enough, than I would use ROC curves or this kind of stuff. $\endgroup$
    – Ubikuity
    Jun 25 at 13:51
  • $\begingroup$ Thanks a lot @Ubikuity, by confidence score I mean probability output of CNN model, probabilities returns by softmax layer. $\endgroup$ Jun 25 at 14:07
  • $\begingroup$ Yup, I would say again that confidence score doesn't seem a great indicator to me, idk why you are using it, accuracy is usually considered the best metric and I've never heard of using confidence score. $\endgroup$
    – Ubikuity
    Jun 25 at 14:16
  • $\begingroup$ @Ubikuity, thanks a lot really helpful. $\endgroup$ Jun 25 at 14:19
1
$\begingroup$

If by confidence score you mean the probability output of your CNN then you have to consider if your models are well calibrated. A well calibrated model is one that a confidence score of, e.g. 0.8 implies 80% accuracy. Another way to think about it is that if you get 100 predictions of class 0 with a confidence score of 0.8, then 80 of those predictions should indeed be of class 0. A specific type of plot, a 'calibration curve', can help you identify that. The above answer the question about the relation between the score and accuracy.

Now, in practical applications, you might not want to use the confidence score as a measure of performance. That is because of effects such as covariate shift and concept drift which have an unpredictable effect on the confidence scores.

$\endgroup$
1
  • $\begingroup$ Thanks alot really helpful @LuckyLuke, by confidence score I mean probability output of my CNN model. $\endgroup$ Jun 25 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.