1
$\begingroup$

Consider a dataset A which has examples for training in a binary classification problem. I have used SVM and applied the weighted method (in MATLAB) since the dataset is highly imbalanced. I have applied weights as inversely proportional to the frequency of data in each class. This is done on training. I have used 10 folds cross-validation for training. After training, I get the confusion matrix on A:

80025 1
0 140

where the first row is for the majority class and the second row is for the minority class. There is only 1 false positive (FP) and all minority class examples have been correctly classified giving true positive (TP) = 140.

PROBLEM: I train again using more data points. Then, I run the trained model on the a new unseen test data set B which was never seen during training. This is the confusion matrix for testing on B .

50075 0
100 0

As can be seen, the minority class has not been classified at all, hence the purpose of weights has failed. Although, there is no FP the SVM fails to capture the minority class examples. I have not applied any weights or balancing method on B. What could be wrong and how to overcome this problem?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Try Resampling Your Dataset

One option is to reduce de occurance of the majority class in your training set.

One other option is to over-sampling the minority class. (you may need to ad a bit of noise )

Other idea: Try Changing Your Performance Metric

The page helped me a lot with unbalance data: Combat Imbalanced Classes in Your Machine Learning Dataset

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your answer. I like the idea of oversampling the minority class but don't understand why to add noise? Can I not simply replicate the exact copies of the minority data? $\endgroup$
    – Ria George
    Dec 13, 2018 at 16:29
  • $\begingroup$ Replicating exact copies of the minority data is not going to help you very much with a model like SVM, since it's just trying to learn how to discriminate between the two classes with a hyperplane. Any unseen negative examples are likely different than the ones in your training set. If you're going to oversample it is probably better to use something like SMOTE, which is referred to in that link. $\endgroup$
    – rabbit
    Dec 13, 2018 at 20:05
  • $\begingroup$ To clarify my previous comment, as I think I was right but for the wrong reasons-- oversampling like this can be helpful in certain situations, but I believe it is equivalent to reweighting your data (as you have already said you've done). However, I think it can also lead to overfitting, especially if the minority class data in your training set is not very representative. Discussed further here. $\endgroup$
    – rabbit
    Dec 13, 2018 at 22:05
1
$\begingroup$

To build on Ludo's answer, and the link they provided: Combat Imbalanced Classes in Your Machine Learning Dataset, you might also try the following:

  • Treating the problem as anomaly detection. There are different supervised and unsupervised ways to do this, but one example would be one-class SVM, as described in this blog post.
  • Using a kernel for the SVM. Is there some high dimensional space that can be used to further distinguish the minority from the majority classes?
  • Trying a different algorithm. Random Forests I believe are supposed to be pretty good for imbalanced datasets, since they try are directly concerned with class entropy.

It's a little difficult to tell what will work the best, so it's worth it to just try a variety of different approaches and see which one works best for your problem.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your answer: I used RBF kernel and supervised SVM. Do you suggest any other kernel that is suitable for imbalanced class? $\endgroup$
    – Ria George
    Dec 13, 2018 at 20:46
  • $\begingroup$ RBF is very flexible and can be prone to overfitting, and also requires choosing a good set of parameters. I'd suggest following the advice here and trying a few different simpler ones (like linear and polynomial) out first. If you underfit your data, perhaps try using more complex ones. $\endgroup$
    – rabbit
    Dec 13, 2018 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.