I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works best with F-Score ~0.83.

Now, I have to apply the trained classifier on an unlabelled dataset DS2 with ~ 5 million entries (and same features). However, for DS2, the target distribution is expected to be highly unbalanced.

Is this a problem? Will the model reproduce the trained target distribution from DS1 when applied on DS2?

If yes, would another algorithm be more robust?

  • 2
    $\begingroup$ I think this may depend quite a bit on why your training data was balanced but your test data is not. $\endgroup$ – Ben Reiniger Mar 5 '19 at 22:07
  • $\begingroup$ @BenReiniger, could you elaborate on that? $\endgroup$ – user3240855 Mar 7 '19 at 6:58
  • $\begingroup$ If you know the real distribution (or have an idea of it) you can use it weight training samples. (You could try using an Inner Product Detector or a Kernel Inner Product Detector if you have the probabilities of the classes, I can provide you with the module but I never used it outside of Computer Vision applications, actually no one ever did, KIPD was published 3 months ago) $\endgroup$ – Pedro Henrique Monforte Apr 6 '19 at 22:12

Is this a problem?

No. not at all.

Will the model reproduce the trained target distribution from DS1 when applied on DS2?

No, not necessarily. If

  1. Balanced set DS1 is a good representative of imbalanced (target) set DS2, and
  2. Classes are well-separated (pointed out by @BenReiniger), which holds easier in higher dimensions,

then model will generate labels with a ratio close to imbalanced DS2 not balanced DS1. Here is a visual example (drawn by myself):

enter image description here

As you see, in the good training set, predictions resemble the real-world ratio, even though model is trained on a balanced set, of course, if classifier does a good job of finding the decision boundaries.

  • 1
    $\begingroup$ As I said elsewhere, I really like this image, but I think it also here illustrates my suggestion about cleanly separable data. Note OP's comment on georg_un's answer: the model really is building in the average response rate. $\endgroup$ – Ben Reiniger Apr 10 '19 at 21:23
  • $\begingroup$ @BenReiniger Thanks for your comment! it helped me think more critically. From OP's comment I could think of the 2nd case where either the training data is not a good representative or classifier is not doing well. About "clearly separable" I agree with you. However, I think the condition becomes easier to satisfy on higher dimensions, which then, classes can be sketched as well-separated areas. For example, there is too much data in image of dogs and cats to tell them apart compared to 1D data. $\endgroup$ – Esmailian Apr 10 '19 at 22:43

For prediction, the GradientBoostingClassifier will only take those features in account that you fed it during training and it will then classify each observation on its own. That means that usually you don't have to worry about the target-distribution of your prediction-dataset, as long you trained your model on a sufficiently extensive training dataset.

  • 1
    $\begingroup$ Hm, my training data has 35% positive target ratio. If applied to DS2, the model finds ~35% positive ratio there as well, even though I'd expect it to be <5% ... $\endgroup$ – user3240855 Mar 7 '19 at 6:58
  • $\begingroup$ A balanced training dataset has a more or less equal distribution of the contained classes. In your case, this would be a distribution of about 50% positive. There are a lot of methods of how to generate an equal distribution by downsampling or oversampling (see this link) $\endgroup$ – georg-un Mar 7 '19 at 11:25
  • $\begingroup$ At the moment there are two possibilities: either your expectation regarding the prediction data set is wrong or the classifier is inaccurate. For this reason, it is common practice to divide the labeled data into a training and a test data set (e.g. in the ratio 70:30 or 80:20) and to examine the performance of the classifier on the test data set. I would recommend doing so and then investigating your results further. Don't forget that you can print out feature importances of the GradientBoostingClassifier with feature_importances_. $\endgroup$ – georg-un Mar 7 '19 at 11:26

A GBM will ultimately try to split your data into rectangular regions and assign each one a constant predicted probability, the proportion of positive training examples in that region. So yes, on the whole the model has baked in the training sample's average response rate.

I think that effect will be lessened if your data is particularly cleanly separable: if each rectangular region is pure, and your test data just happens to be more heavily inclined toward the negative regions, then it will naturally get closer to "the right" answer.

I'm not sure about other models that would be more robust in this way...an SVM probably, not being naturally probabilistic in the first place.

If your context is downsampling, logistic regression has a well-known adjustment for exactly this problem. The same adjustment (to log-odds) seems likely to help in the GBM context as well, though I'm not aware of any analysis to back it up.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.