0
$\begingroup$

There are quite a few questions regarding the optimisation of binary threshold in a classification problem. However, I haven't found a single end-to-end solution to this problem.

In an existing project, I have come up with the following pipeline to train a binary classifier:

  1. Outer-CV due to small to moderate data size.
    1. Inner-CV to tune hyperparameters
    2. Train model with tuned hyperparameters on outer-cv trainset
    3. Predict on the outer-cv test set
    4. Find optimal threshold using prediction probabilities
    5. Get score converting prediction probabilities to class with the optimal threshold
  2. Report avg/std scores along with thresholds

Since there's tiny to no deviation on the score across different folds. (However, the optimal threshold stddev is 3.2)

  1. Tune hyperparameter on entire data
  2. Train model with tuned hyperparameters on entire data

Now my questions are:

  1. Is this pipeline reasonable/correct? i.e., have I missed anything or parts are unnecessary?
  2. How to get the final optimal threshold for my model when predicting in production.
$\endgroup$
2
  • 1
    $\begingroup$ It’s important to understand how to use the probability outputs before you start setting hard thresholds to partition the output space. Vanderbilt’s Frank Harrell has two good blog posts about this topic. fharrell.com/post/class-damage fharrell.com/post/classification $\endgroup$
    – Dave
    Mar 30 at 10:43
  • $\begingroup$ I've read this and I agree with it when it comes to consuming model predictions. However, this doesn't help much in building a robust pipeline to train a model. How do you evaluate a model on a specific metric if you don't convert probabilities to classes? My question is around building a robust pipeline in the scenario that class prediction is a project requirement. $\endgroup$
    – lml
    Mar 30 at 11:43

2 Answers 2

0
$\begingroup$

One way to find a useful threshold in production is to test different possible thresholds in production. If possible, create a multi-arm bandit setup where different thresholds are evaluated on the actual data using the most relevant evaluation metric.

$\endgroup$
1
  • $\begingroup$ This could be an option but in my case the ground truth data and decision have at least 6 months lag $\endgroup$
    – lml
    Mar 31 at 21:42
0
$\begingroup$

Pipeline seems fine , but using two CV may be very time consuming and overkill.

If you had some testing data it would be very easy to decide threshold on whatever optimises your cost function. I think one startegy can be before rolling out to customers wait for some time to generate more testing data with labels and decide the threhold which optimises model performace on those.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.