Say I've built a (completely unrealistic) classification model in Keras that gives me 1.00 accuracy.
And next, I would like to use my model on some new, unseen data, and use
model.predict_proba to get a probability that the observation belongs to class "A". Say this returns to me a 0.75.
Am I interpreting this correctly in English: "100 percent of the time, the model is confident that this new observation is 75 percent likely to be class A" ?
If this is correct, then let's consider if my model was not totally perfect, like in real life, and instead it gave me a 0.40 accuracy. Say my
predict_proba is still 0.75. Then, is this correct:
"40 percent of the time, the model is confident that this new observation is 75 percent likely to be class A." ?
If so...this makes it seem like
predict_proba() is not tell a complete story.
I could mislead someone (say a journalist...or a judge, whoever) by saying, "There's a 75 percent chance this unseen observation belongs to class A"...and that might sound great, if I fail to reveal that this statment was based on a model that had a low accuracy like 0.40.
Am I stating this correctly, and does my apprehension have validity?