Consider a dataset A
which has examples for training in a binary classification problem. I have used SVM and applied the weighted method (in MATLAB) since the dataset is highly imbalanced. I have applied weights as inversely proportional to the frequency of data in each class. This is done on training. I have used 10 folds cross-validation for training. After training, I get the confusion matrix on A
:
80025 1
0 140
where the first row is for the majority class and the second row is for the minority class. There is only 1 false positive (FP) and all minority class examples have been correctly classified giving true positive (TP) = 140.
PROBLEM: I train again using more data points. Then, I run the trained model on the a new unseen test data set B
which was never seen during training. This is the confusion matrix for testing on B
.
50075 0
100 0
As can be seen, the minority class has not been classified at all, hence the purpose of weights has failed. Although, there is no FP the SVM fails to capture the minority class examples.
I have not applied any weights or balancing method on B
. What could be wrong and how to overcome this problem?