35
$\begingroup$

I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test samples as a majority class.

Some good answers about sub-sampling and weighted random forest are given here: What are the implications for training a Tree Ensemble with highly biased datasets?

Which classification methods besides RF can handle the problem in the best way?

Improve this question
$\endgroup$
1

4 Answers 4

Reset to default
24
$\begingroup$
  • Max Kuhn covers this well in Ch16 of Applied Predictive Modeling.
  • As mentioned in the linked thread, imbalanced data is essentially a cost sensitive training problem. Thus any cost sensitive approach is applicable to imbalanced data.
  • There are a large number of such approaches. Not all implemented in R: C50, weighted SVMs are options. Jous-boost. Rusboost I think is only available as Matlab code.
  • I don't use Weka, but believe it has a large number of cost sensitive classifiers.
  • Handling imbalanced datasets: A review: Sotiris Kotsiantis, Dimitris Kanellopoulos, Panayiotis Pintelas'
  • On the Class Imbalance Problem: Xinjian Guo, Yilong Yin, Cailing Dong, Gongping Yang, Guangtong Zhou
Improve this answer
$\endgroup$
20
$\begingroup$

Undersampling the majority class is usually the way to go in such situations.

If you think that you have too few instances of the positive class, you may perform oversampling, for example, sample 5n instances with replacement from the dataset of size n.

Caveats:

  • Some methods may be sensitive to changes in the class distribution, e.g. for Naive Bayes - it affects the prior probabilities.
  • Oversampling may lead to overfitting
Improve this answer
$\endgroup$
2
  • $\begingroup$ Or maybe try some clustering algorithm and use the cluster centres? $\endgroup$
    – Leela Prabhu
    Commented Feb 1, 2017 at 8:18
  • $\begingroup$ You could check this link for oversampling and other methods to deal with imbalanced datasets. $\endgroup$
    – janpreet singh
    Commented Jun 9, 2017 at 13:22
14
$\begingroup$

Gradient boosting is also a good choice here. You can use the gradient boosting classifier in sci-kit learn for example. Gradient boosting is a principled method of dealing with class imbalance by constructing successive training sets based on incorrectly classified examples.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ My understanding is that gradient boosting suffers from the same limitations as RF when dealing with imbalanced data: sci2s.ugr.es/keel/pdf/algorithm/articulo/… $\endgroup$
    – charles
    Commented Sep 15, 2014 at 13:02
  • 1
    $\begingroup$ Boosting is an additional step you take in building the forest that directly addresses imbalance. The paper you link notes this in the intro stating boosting helps even in cases where there is no imbalance. And that paper concludes boosting significantly helps. So not sure where the equivalence between RF and boosting is shown there? $\endgroup$
    – cwharland
    Commented Sep 15, 2014 at 13:09
1
$\begingroup$

In addition to the answers posted here, if the number of positive examples are way too small when compared to the negative examples, then it comes close to being an anomaly detection problem where the positive examples are the anomalies.

You have a whole range of methods for detecting anomalies ranging from using multivariate gaussian distribution to model all the points and then picking those that are 2 or 3 stds away from the mean.

Another food for thought - I have seen quite a few people who randomly sample the negative examples with more examples so that both the classes are same in number. It totally depends on the problem at hand, whether we want them to be balanced or not.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.