# Poor performance of SVM after training for rare events

I found out that Weighted SVM is a classification approach to handle class imbalance problem. My data set is highly imbalanced with rare event (minority class, labeled as 1) and the majority class (label 0). So I implemented the supervised classification weighted svm technique with stratified cross-validation as these are able to handle class imbalance. I added an additional tuning for the C parameter (boxconstraint). The training is done using 5 folds cross-validation approach. The method works well on the training set. I get good performance after the training. This I can say because by looking at the confusion matrix after the training.

cmMatrix =

        1443          27
0          30


It is generally recommended to re-train using the optimized hyperparameters. So, I ran the trained model on the entire dataset again (re-trained it) an predicted on the same dataset.

PROBLEM: If I give a highly imbalanced unseen new data set (this set is never used by the model and is the Test set) to the trained SVM model, the prediction on this data is totally biased towards the majority class as shown below

cmMatrix_TestData =

98     2
5     0


Where did I go wrong? Please help, I practically have no method working for class imbalance problem whereas several articles and suggestion suggest these two approaches which I am not able to make it work for me.

• Are you using cross-validation to train 5 models and then classify based on the output of each model? Or are you using cross-validation as an evaluation technique? – Valentin Calomme Jul 11 '18 at 17:23
• I am using cross-validation to train the model and at the end I have only one model. Inside each fold I am fitting the data to the model and tuning the hyperparameter using another cross-validation loop. – Srishti M Jul 11 '18 at 17:29
• Did you stratified the split when you do perform your cross validation. – The Lyrist Jul 11 '18 at 21:21
• @TheLyrist: I don't know the stratified method. I used  kIdxC = crossvalind('Kfold', length(trainTarg), kFolds); in the hyperparameter tuning. Can you please help how stratified split can be done in Matlab and then I will see if it eliminates the problem or not. – Srishti M Jul 11 '18 at 22:27
• However in the training I have ensured that there is atleast one representative of the minority class by using the line rarray = randperm(bClass )+aClass ; where bClass variable denotes the number of examples in the minority class and aClass variable denotes the number of examples in the majority class. – Srishti M Jul 11 '18 at 22:44

• Alright. The overall accuracy is pretty much meaningless with such an unbalanced problem. You should use the F1 measure as performance. Especially when you optimize for $C$ – Valentin Calomme Jul 11 '18 at 19:31