4
$\begingroup$

I am trying to solve a classification problem on a highly imbalanced data set. I am using SMOTE to over sample the minority samples and down sample the majority ones. After creating a balanced data set, I applied the random forest model. But, the prediction error for the minority class is extremely high even after using a balanced data set. What could possibly be going wrong?

library(DMwR)
new.data <- SMOTE(Clicked ~ ., train, perc.over = 600, perc.under =  80)
table(new.data$Clicked)

rand.forest <- randomForest(Clicked ~., data=new.data, mtry = 7,
                            importance = TRUE, proximity=TRUE, ntree = 1000
                            )


#confusion matrix
table(yhat.rf, test$Clicked)

yhat.rf   0   1
      0 889  47
      1  57   7
Improve this question
$\endgroup$

4 Answers 4

Reset to default
2
$\begingroup$

Balancing your dataset does not guarantee an even prediction split. Imagine the case where your features cannot separate between positive and negative examples at all. In this case, even if you balance the dataset, you will learn a decision boundary that essentially randomly guesses on each example. You would therefore expect that your prediction error would mirror the distribution of the majority/minority classes.

In this scenario you might but have strongly predictive features or you may have insufficient data.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for the answer. So in this case how does one go about improving the recall? $\endgroup$
    – Zack
    Commented Nov 14, 2015 at 6:31
  • $\begingroup$ It is unlikely that there is a straightforward answer in this case, unless you know something is wrong with your prediction (you aren't normalizing the same way at test time, say). You might want to try visualizing your data to convince yourself your features are meaningful. If you find that your features don't correlate well with the output and that simple solutions don't work, your feature space is likely bit informative enough for prediction. $\endgroup$
    – jamesmf
    Commented Nov 14, 2015 at 15:19
2
$\begingroup$

In my experience, giving weights to observations (if the algorithm in use supports it) generally works better for highly imbalanced classification problems. Since, your are using RandomForests I would suggest you to try that.

Improve this answer
$\endgroup$
1
  • $\begingroup$ If for giving weights means setting prior probability for each class in Random Forests, aren't you biasing your classifier? Setting priors is based on the assumption the the prevalence of each class in the training set is the same in the testing set, which is not always the case. $\endgroup$
    – gc5
    Commented Nov 11, 2016 at 14:28
1
$\begingroup$

SMOTE is not designed to work with severe data imbalance specially if you have wide variation within the minority class Try borderline SMOTE Or SMoteBoosting

Improve this answer
$\endgroup$
1
$\begingroup$

By experience, I would also consider to check the ROC and AUC. One might try to use under-sampling as well as other over-sampling methods. In R, you have this toolbox that can provide you different options.

You can also check this paper which provides a comparison between different methods and draw some insights when over- or under-sampling are preferable.

However, I would agree with jamesmf to check the discriminative power of your feature at first.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.