I have recently used xgboost to conduct binary classification in an nlp problem. The idea was to identify if a particular article belonged to an author or not, pretty standard exercise.
The results are outputted as a probability between 0 and 1, and there is the ocasional article that is completely misclassified.
I would like to know if there is a statistical approach that gives me a confidence interval for the probability outputs (for example if I consider all articles with prediction of 0.4 I will get 95% of the articles that belong to the author), or something that helps me make decisions regarding the cut-offs.