Train set and Test set For result and conclusion

Question

I have performed a Logistic regression on a binary classification dataset. The result are as follow :

The training-set accuracy score is 0.8523 while the test-set accuracy to be 0.8442.

For Model evaluation and improvement using Kfold and GridSearch cv :

kfold validation

Applying 5-Fold Cross Validation

from sklearn.model_selection import cross_val_score
​
scores = cross_val_score(model, X_test, y_test, cv = 5, scoring='accuracy')
​
print('Cross-validation scores:{}'.format(scores))
Cross-validation scores:[0.83913352 0.84428267 0.84872159 0.8460309  0.84123601]

We can summarize the cross-validation accuracy by calculating its mean.

Compute Average cross-validation score
print('Average cross-validation score: {:.4f}'.format(scores.mean()))
Average cross-validation score: 0.8439

Original model score is found to be 0.8523. The average cross-validation score is 0.8518. So, we can conclude that cross-validation does not result in performance improvement.

Hyperparameter Optimization using GridSearch CV

from sklearn.model_selection import GridSearchCV
parameters = [{'penalty':['l1','l2']}, 
              {'C':[1, 10, 100, 1000]}]
grid_search = GridSearchCV(estimator = model,param_grid = parameters,scoring = 'accuracy',cv = 5,verbose=0)
grid_search.fit(X_train, y_train)
GridSearchCV(cv=5,
             estimator=LogisticRegression(random_state=0, solver='liblinear'),
             param_grid=[{'penalty': ['l1', 'l2']}, {'C': [1, 10, 100, 1000]}],
             scoring='accuracy')

Examine the best model

Best score achieved during the GridSearchCV

   print('GridSearch CV best score : {:.4f}\n\n'.format(grid_search.best_score_))
    GridSearch CV best score : 0.8520

Print parameters that give the best results

print('Parameters that give the best results :','\n\n', (grid_search.best_params_))
Parameters that give the best results : 

 {'C': 10}

Print estimator that was chosen by the GridSearch

print('\n\nEstimator that was chosen by the search :','\n\n', (grid_search.best_estimator_))

Estimator that was chosen by the search :

 LogisticRegression(C=10, random_state=0, solver='liblinear')
Calculate GridSearch CV score on train set
est
print('GridSearch CV score on test set: {0:0.4f}'.format(grid_search.score(X_test, y_test)))
GridSearch CV score on test set: 0.8446

GridSearch CV score on test set: 0.8525

I have used train set for kfold and gridsearch .

My concern is about which set is taken as for result Train or Test .

Answer 1

Having close values for training-set accuracy and test-set accuracy is good. It means your model is not overfitting. But maybee you may still improve it.

If you do Hyperparameter Optimization using GridSearch CV, you should have:

• train set / validation set with kfold to use during the parameters optimisation only

• test set is for final performance evaluation

To evaluate the final performance, use the values obtained with the test set. It is like if you evaluate your model with new/never seen data.

Moreover in your question: Cross Validation is not used to improve the model but just to have a mean accuracy value which should be more reliable than a single accuracy calculus.

And you should compare final test set score values of both methods. Non optimized score : 0.8442 or 0.8439 against optimized score : 0.8525