There are quite a few questions regarding the optimisation of binary threshold in a classification problem. However, I haven't found a single end-to-end solution to this problem.

In an existing project, I have come up with the following pipeline to train a binary classifier:

  1. Outer-CV due to small to moderate data size.
    1. Inner-CV to tune hyperparameters
    2. Train model with tuned hyperparameters on outer-cv trainset
    3. Predict on the outer-cv test set
    4. Find optimal threshold using prediction probabilities
    5. Get score converting prediction probabilities to class with the optimal threshold
  2. Report avg/std scores along with thresholds

Since there's tiny to no deviation on the score across different folds. (However, the optimal threshold stddev is 3.2)

  1. Tune hyperparameter on entire data
  2. Train model with tuned hyperparameters on entire data

Now my questions are:

  1. Is this pipeline reasonable/correct? i.e., have I missed anything or parts are unnecessary?
  2. How to get the final optimal threshold for my model when predicting in production.
  • 1
    $\begingroup$ It’s important to understand how to use the probability outputs before you start setting hard thresholds to partition the output space. Vanderbilt’s Frank Harrell has two good blog posts about this topic. fharrell.com/post/class-damage fharrell.com/post/classification $\endgroup$
    – Dave
    Mar 30, 2022 at 10:43
  • $\begingroup$ I've read this and I agree with it when it comes to consuming model predictions. However, this doesn't help much in building a robust pipeline to train a model. How do you evaluate a model on a specific metric if you don't convert probabilities to classes? My question is around building a robust pipeline in the scenario that class prediction is a project requirement. $\endgroup$
    – lml
    Mar 30, 2022 at 11:43
  • $\begingroup$ You evaluate on a metric on a metric that inputs probability values instead of classes. Options include log loss (crossentropy loss in some circles) and Brier score. Converting to classes requires an assessment of the damage caused by misclassifications, which might be unknown or dependent on outside factors. // The links I posted in March get into the importance of considering predicted probabilities. $\endgroup$
    – Dave
    Sep 11, 2022 at 20:59

3 Answers 3


One way to find a useful threshold in production is to test different possible thresholds in production. If possible, create a multi-arm bandit setup where different thresholds are evaluated on the actual data using the most relevant evaluation metric.

  • $\begingroup$ This could be an option but in my case the ground truth data and decision have at least 6 months lag $\endgroup$
    – lml
    Mar 31, 2022 at 21:42

Pipeline seems fine , but using two CV may be very time consuming and overkill.

If you had some testing data it would be very easy to decide threshold on whatever optimises your cost function. I think one startegy can be before rolling out to customers wait for some time to generate more testing data with labels and decide the threhold which optimises model performace on those.


You need some form of untouched data to give a true representation of how your pipeline will perform in production. Once you use your holdout set to tune the threshold, you no longer have a holdout set.

Use your CV on your train data to tune this threshold - do final checks on holdout.

I would encourage you to plot your thresholds/probabilities - then fine-tune the threshold to solve your business problem.

  • For example, if you really don't want False positives - choose your metrics appropriately -> and therefore your thresholds.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.