0
$\begingroup$

I'm using the following code for accuracy score calculation. Why is it so that the default configuration gives better result than GridSearch?

Default configuration

clf = svm.SVC(kernel='rbf', gamma='auto')               
clf.fit(x_train, y_train.values.ravel())                

y_train_pred = clf.predict(x_train)             
y_test_pred = clf.predict(x_test)               

print('Train set accuracy: '+'{}'.format(metrics.accuracy_score(y_train, y_train_pred)))                
print('Test set accuracy: '+'{}'.format(metrics.accuracy_score(y_test, y_test_pred)))   

Train set accuracy: 0.861101243339254
Test set accuracy: 0.8480113636363636

GridSearch configuration

param_grid = {'C': (0.001, 0.01, 0.1, 1, 10),
              'kernel': ('linear', 'poly', 'rbf', 'sigmoid'),
              'class_weight': ('balanced', None),
              'gamma' : ('scale', 'auto'),
              'shrinking': (True, False)}

grid_search = GridSearchCV(svm.SVC(gamma='scale'), param_grid, cv=5)
grid_results = grid_search.fit(x_train, y_train.values.ravel())

print(grid_results.best_score_)
print(grid_results.best_estimator_)
print(grid_results.best_params_)

0.8373001776198934
SVC(C=1, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False)
{'C': 1, 'class_weight': None, 'gamma': 'auto', 'kernel': 'rbf', 'shrinking': True}

$\endgroup$
1
$\begingroup$

Technically: Because grid search creates subsamples of the data repeatedly. That means the SVC is trained on 80% of x_train in each iteration and the results are the mean of predictions on the other 20%.

Theoretically: Because you conflate the questions of hyperparameter tuning (selection) and model performance estimation. GridSearch is used for selecting a combination of hyperparameters, performance estimation has not yet happened. The only comparison you should be making is between the parameter combinations within the CV itself (grid_results.cv_results). In my opinion, the reported CV train accuracy is within acceptable boundaries from non-CV training (meaning your SVC is able to extract a lot of generalization from subsamples). See e.g Cawley 2010

It would be interesting to see the reported performance on x_test on the retrained estimator grid_results.best_estimator_.predict(x_test), if it was kept separate from x_train, and how it is different from the first results.

$\endgroup$
  • $\begingroup$ @Tauno Indeed the winning model has the same parameters as the one you trained first. If you are interested in attempting to tune further consider values of C around 1. $\endgroup$ – ludan Jun 30 at 9:58
0
$\begingroup$

Reported performance on x_test on the retrained estimator:

y_test_pred_GS = grid_results.best_estimator_.predict(x_test)
print('Accuracy after GridSearch: '+'{}'.format(metrics.accuracy_score(y_test, y_test_pred_GS))) 

Accuracy after GridSearch: 0.8480113636363636

So, I then interpret this as the default results are good enough and hyperparameter tuning can't make it better.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.