0
$\begingroup$

I have performed a Logistic regression on a binary classification dataset. The result are as follow :

The training-set accuracy score is 0.8523 while the test-set accuracy to be 0.8442.

For Model evaluation and improvement using Kfold and GridSearch cv :

kfold validation

Applying 5-Fold Cross Validation

from sklearn.model_selection import cross_val_score
​
scores = cross_val_score(model, X_test, y_test, cv = 5, scoring='accuracy')
​
print('Cross-validation scores:{}'.format(scores))
Cross-validation scores:[0.83913352 0.84428267 0.84872159 0.8460309  0.84123601]

We can summarize the cross-validation accuracy by calculating its mean.

Compute Average cross-validation score
print('Average cross-validation score: {:.4f}'.format(scores.mean()))
Average cross-validation score: 0.8439

Original model score is found to be 0.8523. The average cross-validation score is 0.8518. So, we can conclude that cross-validation does not result in performance improvement.

Hyperparameter Optimization using GridSearch CV

from sklearn.model_selection import GridSearchCV
parameters = [{'penalty':['l1','l2']}, 
              {'C':[1, 10, 100, 1000]}]
grid_search = GridSearchCV(estimator = model,param_grid = parameters,scoring = 'accuracy',cv = 5,verbose=0)
grid_search.fit(X_train, y_train)
GridSearchCV(cv=5,
             estimator=LogisticRegression(random_state=0, solver='liblinear'),
             param_grid=[{'penalty': ['l1', 'l2']}, {'C': [1, 10, 100, 1000]}],
             scoring='accuracy')

Examine the best model

Best score achieved during the GridSearchCV

   print('GridSearch CV best score : {:.4f}\n\n'.format(grid_search.best_score_))
    GridSearch CV best score : 0.8520

Print parameters that give the best results

print('Parameters that give the best results :','\n\n', (grid_search.best_params_))
Parameters that give the best results : 

 {'C': 10}

Print estimator that was chosen by the GridSearch

print('\n\nEstimator that was chosen by the search :','\n\n', (grid_search.best_estimator_))

Estimator that was chosen by the search :

 LogisticRegression(C=10, random_state=0, solver='liblinear')
Calculate GridSearch CV score on train set
est
print('GridSearch CV score on test set: {0:0.4f}'.format(grid_search.score(X_test, y_test)))
GridSearch CV score on test set: 0.8446

GridSearch CV score on test set: 0.8525

I have used train set for kfold and gridsearch .

My concern is about which set is taken as for result Train or Test .

$\endgroup$
1
  • 1
    $\begingroup$ Cross Validation is not used to improve the model but just to have a mean accuracy value which should be more reliable than a single accuracy calculus. You should compare final test set score values of both methods. Non optimized score : 0.8442 or 0.8439 against optimized score : 0.8525 $\endgroup$
    – Malo
    Aug 23, 2021 at 21:44

1 Answer 1

2
$\begingroup$

Having close values for training-set accuracy and test-set accuracy is good. It means your model is not overfitting. But maybee you may still improve it.

If you do Hyperparameter Optimization using GridSearch CV, you should have:

  • train set / validation set with kfold to use during the parameters optimisation only

  • test set is for final performance evaluation

To evaluate the final performance, use the values obtained with the test set. It is like if you evaluate your model with new/never seen data.

Moreover in your question: Cross Validation is not used to improve the model but just to have a mean accuracy value which should be more reliable than a single accuracy calculus.

And you should compare final test set score values of both methods. Non optimized score : 0.8442 or 0.8439 against optimized score : 0.8525

$\endgroup$
1
  • $\begingroup$ Sir I have put some codes . Please review it . I have used test set for both . $\endgroup$ Aug 23, 2021 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.