What could go wrong if I sample before classification?

Question

I have a million entries in a table that I can use to train a binary classifier. Only 30 thousand of them are positive. Is there anything fundamentally wrong with selecting around 30 thousand negative cases uniformly and then training a binary classifier on only 60 thousand instead of a million entries?

My own reaction to this thought is that underneath it is some assumption about the distribution of the distance between negative and positive. Imagine that the data was just a real number and all that was needed was a cut off. And suppose that cut of was zero. Then if the negative cases were fat tailed, maybe the classifier would choose a sub-optimal negative cut off due to having seen positives near zero but not the rare cases where the negatives were close to zero.

But, that type of problem should be detectable by doing statistical tests on the data.

What else am I missing from this free-lunch scenario?

Answer 1

Search for imbalanced data here and on cross-validated. There is a lot of discussion on techniques, pros and cons.

Lets say it depends. But using "wrong" in the title may be too strong. Something may not be wrong but it may be non-optimal to your use case.

If the model is supposed to generate a well-calibrated probability, a down/up sample may effect the calibration. There are formulas and techniques that may be able to adjust, somewhat. If you do not care about a well-calibrated, then maybe no harm.

The model is losing information. This may be OK if the information is duplicated or all very much on one end of skewed data. But otherwise, the model may be weaker. Of course the model may be stronger if noise was eliminated or irrelevant records were eliminated.

Like you mentioned, the cut-off value may be applied wrong due to loss of information. Perhaps you are doing credit modeling and with the smaller sample, transaction amount became less relevant. But when the model moves into production and see the entire population for scoring, transaction amount is very relevant and the cut off value did not take that into enough consideration.

Or perhaps transaction amount became stronger in the sample and everything turns out better.

If you are interested in interpretation of the model - say coefficients of a logistic regression, then sampling often increases the standard error of the coefficients. If there are suppressors (partial correlation) weakening these may also weaken the coefficients or the model.

I am not sure what the free lunch is. If you down sample, there may or may not be an effect. There may or may not be an effect that you care about. But some work needs to be done there. Trading training time of the computer for analysis time of the human. But part of the analysis may be boot-strapping and comparing so perhaps no training time is eliminated.

You might want to state your goal and reason for down sampling. And read the imbalanced data posts since that may help focus the question.

Answer 2

One possible way to mitigate sampling issues is to apply ensembling. Bootstrap aggregating / bagging could be used to repetitively draw 30K positive samples. Build a model on each 30k positive sample and then aggregate the separate model predictions.

The result of bagging could be a robust classifier that learned the relative contributions of all of the data.

Answer 3

Note: I would appreciate it if someone could tell me why this answer was downvoted. I am here to learn, but not entirely inexperienced, and did think about it. If I am mistaken or unclear - I would like to hear in what way.

---

Whether something goes "wrong" is a matter of what is the goal of the process. But, in a strong sense nothing has to go wrong at all. The question is more - what is the effect?

Suppose that there is a collection of things that fall into several classes. Each thing has several real-valued properties that have a multi variate Gaussian distribution. The mean and variance of the properties are different in each class. A collection of property tuples is chosen by choosing a class and then choosing the properties.

For expositional simplicity think of them having two properties and placing themselves on a plane. There is conditional probability at each point on the plane that a point there will be in each of the classes. Arguably, the best guess for the class of a thing measured to be at some point on the plane is the class with the highest conditional probability. This gives something like a Voronoi diagram for the classes.

But, the shape of this diagram is affected by the prior choice of the probability of the classes. In a sampled population the probability that the trained classifier is responding to is the frequency in the population.

So - if there are only 10 percent of the population in a given one of two classes, and you sample equal numbers from each class - then you bias the classifier to be more likely to choose the 10-percent class.

Sanity Check -- if the positives comprise the entire training set then the model would latch onto just calling everything positive. If the negatives comprise the entire training set then the model would latch onto just calling everything negative. Between, we can expect some fraction of positive and negatives to be guessed.

Whether this is right or wrong depends on whether it is more important to pick this class correctly when it is the right class - or more important to avoid picking it incorrectly.