I'm thinking of a way to build an extension to a binary classifier (actually I will get the output probabilities like in logistic regression, so technically you should call this regression) that outputs a confidence score about how "sure you are that this is the right prediction". I read about conformal quantile prediction and that achieves the scores with an 1-alpha confidence interval, I desire to get the opposite, given the confidence interval or the prediction, how much sure you are that the prediction is correct? For example, to put it in a fancy way: "Since we have a lot of observations similar to this one that you are trying to predict, you should be very confident in your prediction" o "We never saw an observation like this one in training data, so you should not rely on this prediction very much". Am I clear?
If you did not get the idea yet. Here's an idea that comes to my mind right now: For example, when it comes to decision trees. You could build a confidence score based on the fact that your observation falls under a leaf with 10000 observations or i.e 20% of the sample (assuming you have 50k for training), so if that is the case you should be more confident than if your predicted observation would fall in a leave that has 100 observations (0.1% of the training sample). My goal is to do the same but for gradient based boosting trees (GBDT). Doing the same for the GBDT could be possible, but when the number of estimators grows it could get more complicated.
Thanks!
