3
$\begingroup$

I run a Support Vector Machines model on part of my train set with following result:

alg = sk.svm.SVC(probability=True, gamma='auto')
cv_results = model_selection.cross_validate(alg, X_pca, labels, cv =4)

enter image description here

but when I am trying to tune the parameters, with following method:

model=sk.svm.SVC()
params = {'C' : [0.01, 0.1, 1, 10],
      'gamma' : [0.1, 1, 'auto'],
      'probability' : [True] 
     }
clf =  GridSearchCV(model, params, cv=2, return_train_score=False).fit(X_pca, labels)
pd.DataFrame(clf.cv_results_).loc[:, ['mean_test_score', 'rank_test_score', 'params']].sort_values(by='rank_test_score')

enter image description here

So not only all results looks scetchy because they are the same. but also in one of the rows I have C:1, gamma:auto and probability: True which is the same parameters as in first table.

I want to also say, that the same logic I am using for the rest of my 15 ML algorithms and only SVM showed this kind of weird behavior. Wondering that maybe I have some stupid mistake in how I create X_pca and labels data table, I copied code from other algorithm and just replaced second code but it gave the same results.

Can you spot something wrong?

$\endgroup$
  • $\begingroup$ one issue i see is the a 4 fold cross validation in SVM (1st) and 2 fold cross validation in grid search SVM (2nd). $\endgroup$ – Mankind_008 May 31 '18 at 22:36
  • $\begingroup$ I did also 4 fold previously on SVM, but haven't saved the result so now I did just 2-fold to print the results faster. The scores were rougly the same. $\endgroup$ – Mateusz Konopelski Jun 1 '18 at 13:21
1
$\begingroup$

I think the problem might be due to the data, as this code:

from sklearn import svm
from sklearn.model_selection import GridSearchCV
import pandas as pd
import numpy as np

X_pca = np.random.rand(100, 2)
labels = X_pca[:, 0] + X_pca[:, 1] > 0.5

model = svm.SVC()
params = {'C' : [0.01, 0.1, 1, 10],
      'gamma' : [0., 0.1, 1, 'auto'],
      'probability' : [True] 
     }
clf =  GridSearchCV(model, params, cv=2, return_train_score=False)
clf.fit(X_pca, labels)

print(pd.DataFrame(clf.cv_results_).loc[:, ['mean_test_score', 'rank_test_score']] \
        .sort_values(by='rank_test_score'))

gives better output:

    mean_test_score  rank_test_score
10             1.00                1
13             1.00                1
14             1.00                1
15             1.00                1
11             0.95                5
0              0.83                6
1              0.83                6
2              0.83                6
3              0.83                6
4              0.83                6
5              0.83                6
6              0.83                6
7              0.83                6
8              0.83                6
9              0.83                6
12             0.83                6

(consider the fact that the results will depend on the seed of the rand function, but with other seeds they are similar and the mean test score changes using different values of the parameters.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I understand that. It looks that my code works on any example besides my data.. However, weird is that I am using a loop to apply different algorithms on the same data and ONLY SVC gives some wrong results. All other methods are correct. I wonder if this is algorithm specific problem that I shouldn't apply it on that particular set of data. $\endgroup$ – Mateusz Konopelski Jun 4 '18 at 12:35
  • $\begingroup$ What kind of data is it? $\endgroup$ – David Masip Jun 4 '18 at 12:39
  • $\begingroup$ Can you share it? $\endgroup$ – David Masip Jun 4 '18 at 12:39
  • $\begingroup$ Originally data comes from here: kaggle.com/c/digit-recognizer/data $\endgroup$ – Mateusz Konopelski Jun 5 '18 at 12:40
  • $\begingroup$ I've also made public my script: kaggle.com/panpluto/gridsearch-svc/code It's slightly different to above because I was trying different things to make it work. Like wrapping SVC in OnevsOneClassifier $\endgroup$ – Mateusz Konopelski Jun 5 '18 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.