I am currently estimating the certainty of a models estimation by running a neural network model with dropout multiple times and looking at the range of values.
The results confuse me.
I can group the data points. The groups that have most errors are the ones which have the smallest range of values. If my interpretation of "confidence" makes sense, then it means the model is confident about data points where it is wrong.
What are your usual approaches to analyze / fix such problems?
I don't have the option to get more data. I doubt can can clean up the data better.
Another idea is to use LIME (Local interpretable model explanations) to get better insights in why it was wrong.