Poor performance of SVM after training for rare events

Question

I found out that Weighted SVM is a classification approach to handle class imbalance problem. My data set is highly imbalanced with rare event (minority class, labeled as 1) and the majority class (label 0). So I implemented the supervised classification weighted svm technique with stratified cross-validation as these are able to handle class imbalance. I added an additional tuning for the C parameter (boxconstraint). The training is done using 5 folds cross-validation approach. The method works well on the training set. I get good performance after the training. This I can say because by looking at the confusion matrix after the training.

cmMatrix =

        1443          27
           0          30

It is generally recommended to re-train using the optimized hyperparameters. So, I ran the trained model on the entire dataset again (re-trained it) an predicted on the same dataset.

PROBLEM: If I give a highly imbalanced unseen new data set (this set is never used by the model and is the Test set) to the trained SVM model, the prediction on this data is totally biased towards the majority class as shown below

cmMatrix_TestData =

        98     2
         5     0

Where did I go wrong? Please help, I practically have no method working for class imbalance problem whereas several articles and suggestion suggest these two approaches which I am not able to make it work for me.

Answer 1

This I can say because I ran the trained model on the entire dataset again (re-trained it) an predicted on the same dataset.

You seem to be making a fundamental mistake here. If you train and test on the same data, your performance will not be representative of how the model can perform on unseen data points.

Make sure that you train and test on different datasets. If the difference in performance persists, make sure that the validation dataset is representative of your training set. If it's not, this might explain why the model is not performing well on the validation set.