1
$\begingroup$

I found out that Weighted SVM is a classification approach to handle class imbalance problem. My data set is highly imbalanced with rare event (minority class, labeled as 1) and the majority class (label 0). So I implemented the supervised classification weighted svm technique with stratified cross-validation as these are able to handle class imbalance. I added an additional tuning for the C parameter (boxconstraint). The training is done using 5 folds cross-validation approach. The method works well on the training set. I get good performance after the training. This I can say because by looking at the confusion matrix after the training.

cmMatrix =

        1443          27
           0          30

It is generally recommended to re-train using the optimized hyperparameters. So, I ran the trained model on the entire dataset again (re-trained it) an predicted on the same dataset.

PROBLEM: If I give a highly imbalanced unseen new data set (this set is never used by the model and is the Test set) to the trained SVM model, the prediction on this data is totally biased towards the majority class as shown below

cmMatrix_TestData =

        98     2
         5     0

Where did I go wrong? Please help, I practically have no method working for class imbalance problem whereas several articles and suggestion suggest these two approaches which I am not able to make it work for me.

$\endgroup$
  • $\begingroup$ Are you using cross-validation to train 5 models and then classify based on the output of each model? Or are you using cross-validation as an evaluation technique? $\endgroup$ – Valentin Calomme Jul 11 '18 at 17:23
  • $\begingroup$ I am using cross-validation to train the model and at the end I have only one model. Inside each fold I am fitting the data to the model and tuning the hyperparameter using another cross-validation loop. $\endgroup$ – Srishti M Jul 11 '18 at 17:29
  • $\begingroup$ Did you stratified the split when you do perform your cross validation. $\endgroup$ – The Lyrist Jul 11 '18 at 21:21
  • $\begingroup$ @TheLyrist: I don't know the stratified method. I used ` kIdxC = crossvalind('Kfold', length(trainTarg), kFolds);` in the hyperparameter tuning. Can you please help how stratified split can be done in Matlab and then I will see if it eliminates the problem or not. $\endgroup$ – Srishti M Jul 11 '18 at 22:27
  • $\begingroup$ However in the training I have ensured that there is atleast one representative of the minority class by using the line rarray = randperm(bClass )+aClass ; where bClass variable denotes the number of examples in the minority class and aClass variable denotes the number of examples in the majority class. $\endgroup$ – Srishti M Jul 11 '18 at 22:44
2
$\begingroup$

This I can say because I ran the trained model on the entire dataset again (re-trained it) an predicted on the same dataset.

You seem to be making a fundamental mistake here. If you train and test on the same data, your performance will not be representative of how the model can perform on unseen data points.

Make sure that you train and test on different datasets. If the difference in performance persists, make sure that the validation dataset is representative of your training set. If it's not, this might explain why the model is not performing well on the validation set.

$\endgroup$
  • $\begingroup$ I have trained in such a way that there the training set does contain examples from the rare event. But in the test set, I gave 0.5% rare events. How do I know that the weighted svm has seen enough data to give a reasonable performance on the test set. $\endgroup$ – Srishti M Jul 11 '18 at 17:42
  • $\begingroup$ What do you use as performance? Accuracy? F1-score? $\endgroup$ – Valentin Calomme Jul 11 '18 at 18:45
  • $\begingroup$ I saw the overall accuracy and the individual class accuracy. Please let me know if you would like to see my Matlab code to check?It will be immensely useful and clear about what the problem is by running the code. $\endgroup$ – Srishti M Jul 11 '18 at 19:11
  • 1
    $\begingroup$ Alright. The overall accuracy is pretty much meaningless with such an unbalanced problem. You should use the F1 measure as performance. Especially when you optimize for $C$ $\endgroup$ – Valentin Calomme Jul 11 '18 at 19:31
  • 1
    $\begingroup$ Not to you, but if you optimize a parameter, it's based on some metric, and therefore, you should use the F1 measure for that purpose. Also, how many instances do you have? Do you have an absolute number? Is it 3, 10, 100? $\endgroup$ – Valentin Calomme Jul 11 '18 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.