When selecting a probability threshold to maximize the F1 score prior to deploying a model (based on the precision-recall curve), should the threshold be selected based on the training or holdout dataset?


1 Answer 1


Ideally, the threshold should be selected on your training set. Your holdout set is just there to double confirm that whatever has worked on your training set will generalize to images outside of the training set.

This is the reason why hyperparameters tuning like GridSearch and RandomizedSearch in python has a cv parameter to cross-validate between different folds of your training set instead of allowing to choose the best parameters based on metric measured using the holdout set.

  • $\begingroup$ Makes sense, Can I also assume that a well fitted model (not over- or under-fitted) will have the same threshold for the training and test datasets? $\endgroup$ Jul 10, 2020 at 18:04
  • $\begingroup$ Yes, also with the assumption that training images and test images are similar. $\endgroup$ Jul 11, 2020 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.