7
$\begingroup$

I have an imbalanced dataset with 88 positive samples and 128575 negative samples. I was reluctant to over/undersample the data since it's a biological dataset and I didn't want to introduce synthetic data. I built a Random Forest Classifier with this original dataset. I got an F1 score of 0 for the positive class. Zero precision. Zero recall. I cross-checked the predictions and test data. The model predicts some positives none of which are actually positive. Worst performance.

So, I tried to oversample the positive class. I upsampled the positives to 1000 samples. To my surprise, the F1 score for this dataset was 0.97, for the positive class. Then I tried lesser samples. I was able to achieve an F1 score of 0.83 with 200 positive samples, which is just 2.25 times of the original positive samples.

I would like to know why this occurs. For 88 samples, F1 score is 0.00 (rounded off to two digits). For 200 samples it's 0.83. There is no data leakage. All the features are engineered. I used imbalanced-learn module for oversampling. Can someone explain why is this difference in performance?

Improve this question
$\endgroup$
4
  • $\begingroup$ Did you try different (reasonable) thresholds for the class predictions? Are your reported scores on a separate test set, and if so was it also upsampled? What are the models' AUC scores? $\endgroup$
    – Ben Reiniger ♦
    Sep 26, 2019 at 2:38
  • $\begingroup$ @BenReiniger No, I did not change the threshold since both precision and recall were good. The reported scores are for the test set. I upsampled the dataset initially and split it into train and test. The AUC for 200 positive samples is 0.84. $\endgroup$
    – Senthamizhan
    Sep 26, 2019 at 7:43
  • $\begingroup$ I meant changing the threshold for a model on the original data. Splitting after upsampling runs the risk of putting copies of the same datapoint into both the train and test sets. $\endgroup$
    – Ben Reiniger ♦
    Sep 26, 2019 at 12:07
  • $\begingroup$ I think that's the problem. Data leakage. But even if I perform SMOTE, the same problem persists. SMOTE creates synthetic points near the actual point, right? The SMOTE is slower in reaching the F1 score than RandomOverSampler (SMOTE takes 600 positive samples to reach 0.8 F1 score). So am I still wrong in assuming SMOTE reduces data leakage? $\endgroup$
    – Senthamizhan
    Sep 27, 2019 at 6:26

3 Answers 3

Reset to default
4
$\begingroup$

As you mentioned in a comment, you are upsampling before splitting the test set, which leads to data leakage; your scores are not to be trusted. The problem is that a given positive sample may be duplicated and then put into both the training and the test set. Especially with tree models, this is very likely to correctly predict that sample in the test set. The story with SMOTE is similar, but as you pointed out, not quite so severe. In SMOTE you're interpolating between positive samples (see image from imb-learn docs), so if some of those points are in the training set and some in the testing set you're still more likely to correctly identify those points.

Instead, you should split first, upsample the training set second. Alternatively, set class weights (this has the benefit of being independent of the split). Now your test set has a different distribution that the training set, so you'll need to adjust the class prediction threshold, or adjust the probability predictions. See e.g. "Convert predicted probabilities after downsampling to actual probabilities in classification?". Part of the question here is whether you want actual estimates of the probabilities, or just care about the class predictions.

There's a serious question about whether resampling techniques are helpful at all. See e.g.
"What is the root cause of the class imbalance problem?"
"When is unbalanced data really a problem in Machine Learning?"
As a first attempt, I would stick with the original data, fit the random forest, and have a look at different thresholds.

In your case, I would worry that 88 positive samples may just not be enough to see a meaningful pattern. (It might be; it depends on how separated the classes are.)

Improve this answer
$\endgroup$
5
  • $\begingroup$ I like this answer, though messing with the test sets distribution will also lead to results that do not reflect the population. Sampling techniques should only be applied to the training set only. The test set should be formed via stratified sampling to maintain the prevalence rate of what we observe in our dataset (i.e. if we only have few positive cases in reality then the test set should also have few positive cases). The point is to expose the model in the training step to more positive cases in hopes that it can classify more true positives (though at the expense of more false positives) $\endgroup$
    – aranglol
    Sep 27, 2019 at 17:54
  • $\begingroup$ @aranglol, of course you are right, I wasn't thinking. Edited. $\endgroup$
    – Ben Reiniger ♦
    Sep 27, 2019 at 19:35
  • $\begingroup$ Thank you for the explanation @BenReiniger. That was the problem, actually, I tried oversampling after splitting the data. The score dropped to 0. Basically what I have is a chemical sequence of drug molecules (SMILES format). It is not wise to add synthetic data at all, since we can't be sure that the molecule formed by the new synthetic point has similar activity of the actual positives. I'm gonna try different feature engg technique and experiment a new model. Can you suggest me a way to know if the data is separable or not? $\endgroup$
    – Senthamizhan
    Sep 28, 2019 at 13:42
  • $\begingroup$ @Senthamizhan, I'm not aware of a model-agnostic way to test separability; just try a few model types, and as you say feature engineering (which can make all the difference). You could use PCA and take the top two dimensions to see how your positive class lie there, but there's no guarantee that the most variance explained lines up with the two classes. And, while it probably won't help improve the model, with such a small positive class I'd recommend stratified k-fold cross-validation instead of a separate fixed test set. Best of luck, and do ask another question if you need to follow up. $\endgroup$
    – Ben Reiniger ♦
    Sep 30, 2019 at 1:20
  • $\begingroup$ Will surely try the suggested, @BenReiniger. Thank you for your help. $\endgroup$
    – Senthamizhan
    Oct 1, 2019 at 7:17
0
$\begingroup$

When you attempt to train your model without sampling- keeping the imbalanced classes, your model is learning that the easiest way to classify the data is to label everything negative. From an accuracy perspective (total number of correctly classified for each class divided by the total number of instances) your model will have an accuracy of $\frac{128487}{128575}$ or 99%. Essentially it extremely underfits your data to be all one class.

Oversampling corrects the imbalance, and makes your algorithm work a little bit harder to figure out the true shape of the data. Lumping everything into one category won't work. You could have also corrected your imbalance by undersampling the negative class. Typically the rule of thumb is undersampling when you have tens of thousands to hundreds of thousands of rows, and oversampling when your data is smaller (tens of thousands or less).

Here is a good reference for dealing with class imbalances in machine learning.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ But the model didn't just predict the negative class all the time. According to the OP, the model does predict positives at a rate in line with the overall prevalence, it's just never correct. $\endgroup$
    – Nuclear Hoagie
    Sep 25, 2019 at 18:34
  • $\begingroup$ Hello Kate, thank you for your answer. I am aware of the effects of class imbalance on the model. I would like to know why the increase in just 112 samples (88 to 200) boosted the F1 score from 0.00 to 0.83. How could such a small increase in samples, compared to negative class, can boost the performance to this extent? $\endgroup$
    – Senthamizhan
    Sep 26, 2019 at 7:46
0
$\begingroup$

If I understand the situation described by OP, the answer is already in the question. It's because OP has an imbalanced training set. With only a double digit number of positive samples out of over 120000, the model would have the most statistical success just always predicting negative all the time.

It is not incorrect to over-sample or under-sample data (biological or otherwise), so re-sampling is a perfectly legitimate solution if done carefully. Repetition works if you really want to avoid synthetic data, but otherwise there are a number of techniques for that as well (e.g. SMOTE).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hello Jason, yes you are absolutely right. For more clarification, I plotted the F1 scores vs the number of positive samples. The score increased with increased in samples. So, I am intrigued by how a mere 112 samples (increase from 88 to 200) can boost the F1 score from 0.00 to 0.83. It'd be helpful to know the reason behind this performance boost. $\endgroup$
    – Senthamizhan
    Sep 26, 2019 at 7:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.