I'm trying to use GridSearchCV to select the optimal C value in this simple SVM problem with non-separable samples. The issue I'm having is that when I run the code the optimal C is chosen to be ridiculously small (~e-18) so that the margin is expanded to contain all samples. Even when I alter the samples so that they are easily separable, the optimal C is still on the scale of e-18. GridSearchCV selects a very small C however I try to alter the samples. Does anyone know why this is happening?

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets.samples_generator import make_blobs
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import GridSearchCV

X, y = make_blobs(n_samples = 500, centers = 2, random_state = 6,
                  cluster_std = 1.2)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.scatter(X[:,0], X[:,1], c = y, cmap = 'rainbow', s = 30,
           edgecolors = 'white')
ax.set_xlabel(r'$x_1$', fontsize = 20)
ax.set_ylabel(r'$x_2$', fontsize = 20)

svc = SVC(kernel = 'linear')
c_space = np.logspace(-20, 1, 50)
param_grid = {'C': c_space}
svc_cv = GridSearchCV(svc, param_grid, cv = 5)
svc_cv.fit(X, y)
c = svc_cv.best_params_['C']
svc.C = c
svc.fit(X, y)

support_vecs = svc.support_vectors_

x1_min = min(X[:,0])
x1_max = max(X[:,0])
x2_min = min(X[:,1])
x2_max = max(X[:,1])
x1 = np.linspace(x1_min, x1_max, 100)
x2 = np.linspace(x2_min, x2_max, 100)
X1, X2 = np.meshgrid(x1, x2)
points = np.vstack([X1.ravel(), X2.ravel()]).T
boundary = svc.decision_function(points).reshape(X1.shape)
ax.contour(X1, X2, boundary, colors = 'k', levels = [-1, 0, 1],
           linestyles = ['--', '-', '--'])
ax.scatter(support_vecs[:,0], support_vecs[:,1], s = 250, linewidth = 1,
           facecolors = 'none', edgecolors = 'k')


Have a look at svc_cv.cv_results_: there are many values of C that tied for best, with accuracy 99.6%, and the chosen C is the smallest of those. The point is that the width of the margin doesn't affect the actual hyperplane very much, and so the accuracy score doesn't change very much.

A few suggestions:

  1. With larger and less-separable data sets, this might get minimized, since small changes in the hyperplane would be more likely to classify points differently.

  2. For this case, where many scores are exactly equal, you might prefer to tie-break for larger values of C. This can be easily accomplished by just reversing the order of the list c_space, or more robustly by defining a custom scorer for the search, that takes the mean test score plus some function of C.

  3. Something more refined than accuracy as the scorer for the search could help separate the values for different C, but it's not clear what would be best. Something like log-loss would require calibrating probabilities; maybe AUROC?

  • $\begingroup$ Thanks for the useful suggestions, I will have a go at implementing them. I was sure I had made a mistake somewhere. $\endgroup$ – Nitram Mar 28 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.