I am training a normal and a balanced Linear SVM using imbalance data, and testing both using F1-score.
(By balanced Linear SVM, I mean that each observation has a weight inverse to its frequency, so that it "oversamples" from the minority class and "undersamples" from the majority class.)
Not surprisingly, using a multi-class dataset and solving it individually as one-vs-rest binary problems, balanced Linear SVM beats unbalanced Linear SVM on every class:
normal vs balanced target 01 - scores: 0.272 vs 0.608 target 02 - scores: 0.391 vs 0.587 target 03 - scores: 0.433 vs 0.546 target 04 - scores: 0.659 vs 0.655 target 05 - scores: 0.000 vs 0.257 target 06 - scores: 0.431 vs 0.475 target 07 - scores: 0.000 vs 0.249 target 08 - scores: 0.000 vs 0.053 target 09 - scores: 0.576 vs 0.155 target 10 - scores: 0.000 vs 0.550 OneVsRest - scores: 0.565 vs 0.540
But when it comes to using a OneVsRest classifier (last row), the unbalanced Linear SVM beats the Linear SVM, using the average of the F1 scores. This happens to me in a lot of datasets.
Here is the code I have used:
# -*- coding: utf-8 -*- from sklearn.svm import LinearSVC from sklearn.multiclass import OneVsRestClassifier from sklearn.cross_validation import StratifiedShuffleSplit from sklearn.metrics import f1_score import numpy as np d = np.loadtxt('yeast.csv', delimiter=',') X = d[:, 0:-1] y = d[:, -1] print ' none vs balanced' for target in np.unique(y): scores =  * 2 k = 6 for tr, ts in StratifiedShuffleSplit(y, k, 0.2): for i, w in enumerate([None, 'balanced']): _y = (y == target).astype(int) yp = LinearSVC(class_weight=w).fit(X[tr], _y[tr]).predict(X[ts]) scores[i] += f1_score(_y[ts], yp) / k print 'target %02d - scores: %.3f vs %.3f' % (target, scores, scores) scores =  * 2 for tr, ts in StratifiedShuffleSplit(y, k, 0.2): for i, w in enumerate([None, 'balanced']): m = OneVsRestClassifier(LinearSVC(class_weight=w)) yp = m.fit(X[tr], y[tr]).predict(X[ts]) scores[i] += f1_score(y[ts], yp, pos_label=None, average='weighted') / k print 'OneVsRest - scores: %.3f vs %.3f' % (scores, scores)
none vs balanced target 01 - scores: 0.241 vs 0.612 target 02 - scores: 0.402 vs 0.604 target 03 - scores: 0.444 vs 0.552 target 04 - scores: 0.666 vs 0.650 target 05 - scores: 0.000 vs 0.286 target 06 - scores: 0.370 vs 0.500 target 07 - scores: 0.000 vs 0.280 target 08 - scores: 0.000 vs 0.082 target 09 - scores: 0.563 vs 0.147 target 10 - scores: 0.000 vs 0.511 OneVsRest - scores: 0.567 vs 0.543
Isn't this surprisingly? I know that OneVsRest has a lot of ties, and so scores will be used, as determined by the distance to the separation hyperplane. Still, why does balanced Linear SVM loses the war when it keeps wining every battle?
EDIT: I think the fact that one model wins pretty much every "battle", but not the "war" is related to the fact that One Vs Rest uses confidence scores, and not the actual classification, which makes sense since it avoids ties, and when score[k1] > score[k2] we can assume it prefers k1 > k2. See wikipedia.