I have a basic, yet quite complex problem to solve right now. Let's say we have a training set of 20,000 samples in my training set, out of which 3 to 4% is flagged as "True", the rest is flagged as "False". I want to train a classifier (typically XGBClassifier or LGBMClassifier are those I worked a bit with).
What I'm currently doing is that I find the best parameters using GridSearchCV. But my objective is to minimize the amount of samples I would flag as "True" once I try on the test set. Should I train the algorithm with a typical F1 metric, and ONLY THEN find the best threshold that suits my needs? Or should I create a custom metric that would implicitly force the algorithm not to flag too many samples as positive?
Hope this is clear!