# How to select 'cutoff' of classifier probability

I have recently used xgboost to conduct binary classification in an nlp problem. The idea was to identify if a particular article belonged to an author or not, pretty standard exercise.

The results are outputted as a probability between 0 and 1, and there is the ocasional article that is completely misclassified.

I would like to know if there is a statistical approach that gives me a confidence interval for the probability outputs (for example if I consider all articles with prediction of 0.4 I will get 95% of the articles that belong to the author), or something that helps me make decisions regarding the cut-offs.

• Itamar's answer is correct. I wanted to add that there is a great video to explain it here: youtube.com/watch?v=OAl6eAyP-yo This video utilizes a neat online website to let you play with it to understand it better: navan.name/roc – Josh Jul 13 at 19:51

Agree with Itama answer. Just want to edit that ROC curve shows trade off between TPR (recall) and FPR (not precision). Using ROC curve, you can find an optimal threshold via finding the point on the curve closest to point (0, 1), meaning we try to make FPR close to 0 while TPR close to 1.