I have a basic, yet quite complex problem to solve right now. Let's say we have a training set of 20,000 samples in my training set, out of which 3 to 4% is flagged as "True", the rest is flagged as "False". I want to train a classifier (typically XGBClassifier or LGBMClassifier are those I worked a bit with).

What I'm currently doing is that I find the best parameters using GridSearchCV. But my objective is to minimize the amount of samples I would flag as "True" once I try on the test set. Should I train the algorithm with a typical F1 metric, and ONLY THEN find the best threshold that suits my needs? Or should I create a custom metric that would implicitly force the algorithm not to flag too many samples as positive?

Hope this is clear!

  • $\begingroup$ Sorry it's not totally clear :) Do you mean that you expect an even smaller proportion of true instances in the test set compared to your training set? Or is it that it's important for your problem to avoid false positive errors? $\endgroup$ – Erwan Oct 1 '19 at 23:07
  • 2
    $\begingroup$ This is a classic example of class imbalance (added tag), which is a huge subtopic in itself with its own dedicated methods and approaches; I suggest you start googling ruthlessly... $\endgroup$ – desertnaut Oct 1 '19 at 23:46
  • $\begingroup$ What I meant is that I'm aiming to keep the same proportion of positives in my test set! Let's say my training set contains 4% positives, then I'd want my algorithm to "guess" about 4% of positives on the test set as well $\endgroup$ – Dylan Oct 2 '19 at 20:03

as desertnaut was saying, I think you are talking about class imbalance.

If you meet class imbalance "in the wild" there are a couple of ways to tackle this. Your approach of creating a custom metric would fall under cost function based approaches. You can use cost functions that handle false negatives as worse than false positives (which is usually the case when we're talking about class imbalance).

Another way would be to use sampling based approaches. These are rather simple and I personally tend to use these more often since it's easier to play around with aka experiment with them. Two extremes that can be combined as you wish:

  1. Boosting one class / Oversampling: Look at the distribution of Class A and Class B. Boost the class you want to have more effect to have more datasamples changing your algorithm.

  2. Undersampling: Look at the distribution of Class A and Class B. Remove datasets of the class you want to have less effect of your algorithm.

In reality I usually go for a combination of both to have more variance in my data. For example boosting Class A by 110% and reducing Class B to 80% to achieve a balanced dataset (although it sounds to me that you want to have an imbalance just the other way round. Take care about bias with that though)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.