0
$\begingroup$

I was playing around with the credit default dataset in UCI ("https://archive.ics.uci.edu/ml/machine-learning-databases/00350/default%20of%20credit%20card%20clients.xls")

These are the steps i have undertaken so far:

  1. Created a Pipeline for Perceptron
  2. Parameter values were provided for learning rate and epochs.
  3. Passed the estimator and param grids to GridSearch to get the best estimator
  4. GridSearch provided me with best score for a particular learning rate and epoch
  5. used predict method on the gridsearch and recalculated accuracy score

Parameters provided for gridsearch

{'perceptron__max_iter': [1,5,8,10], 'perceptron__eta0': [0.5,.4, .2, .1]}

Question: The best parameters provided by GridSearch are not really the best parameters. I am not getting same scores using predict method on the gridsearch. Why is this the case? WHat am i doing wrong?

So acc to gridsearch best param are : {'perceptron__eta0': 0.5, 'perceptron__max_iter': 8}

Accuracy score : 0.7795238095238095

However if i use these best parameters and call predict on gridsearch gives a totally different value, accuracy score dips to 0.5882222222222222

Please find code below.

import pandas as panda


from sklearn.model_selection import learning_curve, train_test_split,GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline, Pipeline
from sklearn.metrics import accuracy_score, mean_absolute_error, classification_report
from sklearn.linear_model import Perceptron, LogisticRegression
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier


from matplotlib import pyplot as plot
import seaborn as sns


from numpy import bincount, linspace, mean, std

remote_location = "https://archive.ics.uci.edu/ml/machine-learning-databases/00350/default%20of%20credit%20card%20clients.xls"
data = panda.read_excel(remote_location,sheet_name = "Data", header = 1)

data.rename(str.lower, inplace = True, axis = 'columns')


_y_target = data['default payment next month'].values

columns = data.columns.tolist()
columns.remove('default payment next month')

_x_attributes = data[columns].values


## meaning of stratify = _y_target. returns test and training data having the same proportions of class label '_y_target'
_x_train,_x_test,_y_train, _y_test = train_test_split(_x_attributes, _y_target, test_size =0.30, stratify = _y_target, random_state = 1)

## lets check the distribution. we can see 4times the lower value as was the case before as well. train/test set distributed well
print("label counts in y train %s" %bincount(_y_train))
print("label counts in y test %s" %bincount(_y_test))
parameter_grid = {'perceptron__max_iter': [1,5,8,10], 'perceptron__eta0': [0.5,.4, .2, .1]}

pipeline = make_pipeline( StandardScaler(),Perceptron(random_state = 1))
gridsearch = GridSearchCV(estimator = pipeline, param_grid = parameter_grid, cv = 10, n_jobs = 1, scoring = 'accuracy')

search = gridsearch.fit(_x_train, _y_train)
print(search.best_params_)
print(search.best_score_)

_y_prediction = gridsearch.predict(_x_test)


print("Accuracy score %s" %accuracy_score(_y_test,_y_prediction))
print("Classification report  \n %s" %(classification_report(_y_test, _y_prediction)))

My output is as below:

{'perceptron__eta0': 0.5, 'perceptron__max_iter': 8}
0.7795238095238095
Accuracy score 0.5882222222222222
Classification report  
              precision    recall  f1-score   support

          0       0.78      0.66      0.71      7009
          1       0.22      0.35      0.27      1991

avg / total       0.66      0.59      0.62      9000
$\endgroup$
1
$\begingroup$

Summarizing your results - your trained a model using gridsearch.
accuracy score on the train set is ~0.78.
accuracy score on the test set is ~0.59.
Rephrasing you questions: why do my model performance on the test set is worse than on my train set?

This phenomena is very common - and I can think of two potential explanations:
1) Overfitting: your trained model had learned the 'noise' in the train set and not the actual pattern.
Then when you use your model to predict on the test set, it predicts the noise he had encountered(which is not relevant for the train set - thus lower accuracy).
2) Train set and data set are not generated from the same process/describe different parts of it. In this case - the pattern learnt by the trained model is not relevant to the test set and accuracy will drop.
This may happen in situations where the train/test split is done without considering the actual underlying process. For example - an image classification problem where you model whether this picture is an animal or not and the train data has only human and dogs sample and the train set has only fish and airplanes pictures. This is less likely to be the explanation, since your using grid search, which adds randomness to this splitting.

| improve this answer | |
$\endgroup$
  • $\begingroup$ thank you for your response. however i am using the same gridsearch instance and calling predict operation on it. according to scikit docs (scikit-learn.org/stable/modules/generated/…), predict method on gridsearch instance will use the same best params . in that case, wouldnt i be getting the same results. infact i get better results when i manually run it using learning rate of 0.1 and 10 epochs. $\endgroup$ – user1906450 Oct 28 '18 at 14:42
  • $\begingroup$ Yes, it uses the same best params. But from what I understand, you use the predict on a different data set - X_test. $\endgroup$ – yoav_aaa Oct 28 '18 at 14:49
  • $\begingroup$ oh ok my bad , i didnt mention the train_test_split part of the code. updated the original question. the class distribution among test set and train set is pretty much the same 1:4. so if i understand your point well, in this particular instance using perceptron model on the data sets leads to overfitting. p.s. i dont see this behavior when i replace perceptron model in the pipeline with other models such as SVM, decisiontrees, etc $\endgroup$ – user1906450 Oct 28 '18 at 14:52
  • $\begingroup$ The more complex the model is the more probable it will overfit. $\endgroup$ – yoav_aaa Oct 28 '18 at 14:59
  • $\begingroup$ hmm.. it actually performs better for logistic regression. i have also plotted the learnnig curve and i can see clear overfit cases for decisiontrees,randomforests. give me a moment , i will share the jupyter notebook. if it makes my code clear. will that be too much to ask? $\endgroup$ – user1906450 Oct 28 '18 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.