I am trying to select a model from different CNN trained models based on some parameter. Initially, I was considering using model confidence score to decide which model is better. Now I am considering using accuracy for the selection of a good model among other models. I need to ask is there a relation between accuracy and confidence score, does it means high accuracy leads to high confidence score and vice versa. Thanks
If by confidence score you mean the probability output of your CNN then you have to consider if your models are well calibrated. A well calibrated model is one that a confidence score of, e.g. 0.8 implies 80% accuracy. Another way to think about it is that if you get 100 predictions of class 0 with a confidence score of 0.8, then 80 of those predictions should indeed be of class 0. A specific type of plot, a 'calibration curve', can help you identify that. The above answer the question about the relation between the score and accuracy.
Now, in practical applications, you might not want to use the confidence score as a measure of performance. That is because of effects such as covariate shift and concept drift which have an unpredictable effect on the confidence scores.