1
$\begingroup$

When selecting a probability threshold to maximize the F1 score prior to deploying a model (based on the precision-recall curve), should the threshold be selected based on the training or holdout dataset?

$\endgroup$
1
$\begingroup$

Ideally, the threshold should be selected on your training set. Your holdout set is just there to double confirm that whatever has worked on your training set will generalize to images outside of the training set.

This is the reason why hyperparameters tuning like GridSearch and RandomizedSearch in python has a cv parameter to cross-validate between different folds of your training set instead of allowing to choose the best parameters based on metric measured using the holdout set.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Makes sense, Can I also assume that a well fitted model (not over- or under-fitted) will have the same threshold for the training and test datasets? $\endgroup$ – thereandhere1 Jul 10 at 18:04
  • $\begingroup$ Yes, also with the assumption that training images and test images are similar. $\endgroup$ – Vincent Yong Jul 11 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.