Balanced Linear SVM wins every class except One vs All

Question

I am training a normal and a balanced Linear SVM using imbalance data, and testing both using F1-score.

(By balanced Linear SVM, I mean that each observation has a weight inverse to its frequency, so that it "oversamples" from the minority class and "undersamples" from the majority class.)

Not surprisingly, using a multi-class dataset and solving it individually as one-vs-rest binary problems, balanced Linear SVM beats unbalanced Linear SVM on every class:

                   normal vs balanced
target 01 - scores: 0.272 vs 0.608
target 02 - scores: 0.391 vs 0.587
target 03 - scores: 0.433 vs 0.546
target 04 - scores: 0.659 vs 0.655
target 05 - scores: 0.000 vs 0.257
target 06 - scores: 0.431 vs 0.475
target 07 - scores: 0.000 vs 0.249
target 08 - scores: 0.000 vs 0.053
target 09 - scores: 0.576 vs 0.155
target 10 - scores: 0.000 vs 0.550
OneVsRest - scores: 0.565 vs 0.540

But when it comes to using a OneVsRest classifier (last row), the unbalanced Linear SVM beats the Linear SVM, using the average of the F1 scores. This happens to me in a lot of datasets.

Here is the code I have used:

# -*- coding: utf-8 -*-

from sklearn.svm import LinearSVC
from sklearn.multiclass import OneVsRestClassifier
from sklearn.cross_validation import StratifiedShuffleSplit
from sklearn.metrics import f1_score
import numpy as np

d = np.loadtxt('yeast.csv', delimiter=',')
X = d[:, 0:-1]
y = d[:, -1]

print '                     none vs balanced'
for target in np.unique(y):
    scores = [0] * 2
    k = 6
    for tr, ts in StratifiedShuffleSplit(y, k, 0.2):
        for i, w in enumerate([None, 'balanced']):
            _y = (y == target).astype(int)
            yp = LinearSVC(class_weight=w).fit(X[tr], _y[tr]).predict(X[ts])
            scores[i] += f1_score(_y[ts], yp) / k
    print 'target %02d - scores: %.3f vs %.3f' % (target, scores[0], scores[1])

scores = [0] * 2
for tr, ts in StratifiedShuffleSplit(y, k, 0.2):
    for i, w in enumerate([None, 'balanced']):
        m = OneVsRestClassifier(LinearSVC(class_weight=w))
        yp = m.fit(X[tr], y[tr]).predict(X[ts])
        scores[i] += f1_score(y[ts], yp, pos_label=None, average='weighted') / k
print 'OneVsRest - scores: %.3f vs %.3f' % (scores[0], scores[1])

As the dataset, I have used here Yeast (UCI), for which a sklearn-ready version can be found here.

Results:

                     none vs balanced
target 01 - scores: 0.241 vs 0.612
target 02 - scores: 0.402 vs 0.604
target 03 - scores: 0.444 vs 0.552
target 04 - scores: 0.666 vs 0.650
target 05 - scores: 0.000 vs 0.286
target 06 - scores: 0.370 vs 0.500
target 07 - scores: 0.000 vs 0.280
target 08 - scores: 0.000 vs 0.082
target 09 - scores: 0.563 vs 0.147
target 10 - scores: 0.000 vs 0.511
OneVsRest - scores: 0.567 vs 0.543

Isn't this surprisingly? I know that OneVsRest has a lot of ties, and so scores will be used, as determined by the distance to the separation hyperplane. Still, why does balanced Linear SVM loses the war when it keeps wining every battle?

EDIT: I think the fact that one model wins pretty much every "battle", but not the "war" is related to the fact that One Vs Rest uses confidence scores, and not the actual classification, which makes sense since it avoids ties, and when score[k1] > score[k2] we can assume it prefers k1 > k2. See wikipedia.

Answer 1

I think the following synthetic example explains what is going on:

from sklearn.base import clone
from sklearn.svm import SVC
import numpy as np
from sklearn.datasets import make_blobs

X, y = make_blobs(1000, centers=[[1, 1], [-1, -1], [1, -1]], cluster_std=0.8)

models = [
    ('normal', SVC()),
    ('balanced', SVC(class_weight='balanced')),
]

import matplotlib.pyplot as plot
plot.ioff()

for i, (name, m) in enumerate(models):
    for k in np.unique(y):
        m = clone(m).fit(X, (y == k).astype(int))

        xx, yy = np.meshgrid(np.arange(-3, 3, 0.1), np.arange(-3, 3, 0.1))
        z = m.predict(np.c_[xx.ravel(), yy.ravel()])
        z = z.reshape(xx.shape)
        colors = ['blue', 'green', 'red']
        plot.contour(xx, yy, z, colors=colors[k])
    plot.scatter(X[:, 0], X[:, 1], c=y)
    plot.title(name)
    plot.show()

The balanced SVM is clearly superior when considering each model individually, at least as measured by the F1 score, which punishes false positives and false negatives equally.

But One-vs-Rest uses maximum score, which in this case is given by the distance to the decision hyperplane. It is not yet clear why this is worse in some cases, but clearly it is possible for one model to beat another for individual classes, but to lose for One-vs-Rest: how well the relative distances are modeled is what counts in One-vs-Rest.