0
$\begingroup$

I have a model with an imbalanced dataset, lets say 5% of the rows are from the positive class. If I resample my data using something like SMOTE, or removing rows from the larger class (downsampling), I can change that imbalance to ~40%.

This works well and the performance in training is good (good performance on train/valid data). But when this model makes predictions in production, the precision is much lower (25% compared with 75% in training) and the recall is much higher (99% compared with 80%). As such, I am wondering whether this is because of the resampling.. The model thinks that the positive case is much more likely to occur than it actually is.

I have checked for data leakage, and cant find any issues. I am also stopping my model once performance plateaus (although many of the predictions are either close to 0 or 100% class probability, which seems odd to me). Any ideas?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Using precision and recall, you have a cutoff value. It sounds like the cutoff value is not optimal for the production. How was the cutoff value set? From a validation data that was not downsampled or from the training set that was? Is the production data now drifting from the training data? One of the risks of downsampling in dev is not giving the model the diversity of data that seen in prod. If the data diversity is high, downsampling or upsampling, may be harmful. Check without any down/up sampling and focus on the cutoff value. $\endgroup$
    – Craig
    Commented Mar 4, 2021 at 11:31
  • $\begingroup$ Good news! Class imbalance is not a problem: stats.stackexchange.com/questions/357466/… $\endgroup$
    – Dave
    Commented Mar 4, 2021 at 11:42
  • $\begingroup$ I'd start by revisiting your validation strategy: Did you apply k-fold CV or hold-out validation? Did you apply sampling techniques only to train data or to validation too? Moreover, I'd inspect ROC or precision/recall curves to better understand the performance of your classifier. $\endgroup$
    – Jonathan
    Commented Mar 4, 2021 at 12:00
  • $\begingroup$ Have you shuffled your data properly? Try K-Fold Cross validation on your dataset and check precision and recall then again. The imbalanced data shouldn't be the issue in such huge discrepency. Shuffle data properly and select random data to train and test. $\endgroup$
    – Felix Tenn
    Commented Mar 4, 2021 at 12:56
  • $\begingroup$ All very good comments, I will get back to you! $\endgroup$
    – Christopher Collins
    Commented Mar 4, 2021 at 17:16

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.