I have performed a Logistic regression on a binary classification dataset. The result are as follow :
The training-set accuracy score is 0.8523 while the test-set accuracy to be 0.8442.
For Model evaluation and improvement using Kfold and GridSearch cv :
kfold validation
Applying 5-Fold Cross Validation
from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X_test, y_test, cv = 5, scoring='accuracy')
print('Cross-validation scores:{}'.format(scores))
Cross-validation scores:[0.83913352 0.84428267 0.84872159 0.8460309 0.84123601]
We can summarize the cross-validation accuracy by calculating its mean.
Compute Average cross-validation score
print('Average cross-validation score: {:.4f}'.format(scores.mean()))
Average cross-validation score: 0.8439
Original model score is found to be 0.8523. The average cross-validation score is 0.8518. So, we can conclude that cross-validation does not result in performance improvement.
Hyperparameter Optimization using GridSearch CV
from sklearn.model_selection import GridSearchCV
parameters = [{'penalty':['l1','l2']},
{'C':[1, 10, 100, 1000]}]
grid_search = GridSearchCV(estimator = model,param_grid = parameters,scoring = 'accuracy',cv = 5,verbose=0)
grid_search.fit(X_train, y_train)
GridSearchCV(cv=5,
estimator=LogisticRegression(random_state=0, solver='liblinear'),
param_grid=[{'penalty': ['l1', 'l2']}, {'C': [1, 10, 100, 1000]}],
scoring='accuracy')
Examine the best model
Best score achieved during the GridSearchCV
print('GridSearch CV best score : {:.4f}\n\n'.format(grid_search.best_score_))
GridSearch CV best score : 0.8520
Print parameters that give the best results
print('Parameters that give the best results :','\n\n', (grid_search.best_params_))
Parameters that give the best results :
{'C': 10}
Print estimator that was chosen by the GridSearch
print('\n\nEstimator that was chosen by the search :','\n\n', (grid_search.best_estimator_))
Estimator that was chosen by the search :
LogisticRegression(C=10, random_state=0, solver='liblinear')
Calculate GridSearch CV score on train set
est
print('GridSearch CV score on test set: {0:0.4f}'.format(grid_search.score(X_test, y_test)))
GridSearch CV score on test set: 0.8446
GridSearch CV score on test set: 0.8525
I have used train set for kfold and gridsearch .
My concern is about which set is taken as for result Train or Test .
