Handling huge dataset imbalance (2 class values) and appropriate ML algorithm

Question

I have train and test sets of chronological data consisting of 305000 instances and 70000,appropriately. There are 15 features in each instance and only 2 possible class values ( NEW,OLD). The problem is that there are only 725 OLD instances in the train set and 95 in the test.

The only algorithm which succeeds for me to handle imbalance is NaiveBayes in Weka (0.02 precision for OLD class), others (trees) classify each instance as NEW. What is the best approach to handle the imbalance and the appropriate algorithm in such a case?

Thank you in advance.

Answer 1

I'm not allowed to comment, but I have more a suggestion: you could try to implement some "Over-sampling Techniques" like SMOTE: http://scholar.google.com/scholar?q=oversampling+minority+classes

Answer 2

You can apply a clustering algorithm to the instances in the majority class and train a classifier with the centroids/medoids offered by the cluster algorithm. This is subsampling the majority class, the converse of oversampling the minority class.

Answer 3

In addition to undersampling the majority class (i.e. taking only a few NEW), you may consider oversampling the minority class (in essence, duplicating your OLDs, but there are other smarter ways to do that)

Note that oversampling may lead to overfitting, so pay special attention to testing your classifiers

Check also this answer on CV:

• https://stats.stackexchange.com/a/108325/49130

Answer 4

In weka, you could assign the weights to your learning instances. Assign the weights in inverse proportion to the class frequency weight and you should be good to go. Another way would be to play around with sampling.