is this grid search methodology correct?

Question

I have some data say 1 million rows, I then put 200k aside (to validate against) and call this remaining 800,000 the training set (X) as you see below, so it is not the entire data and the remaining 200k is the validation set. My question is, is the below grid search ok? My roc curves were extremely high say 0.96. I suspect it is because of the content below, should I instead use grid search on x_train and y_train instead then further split x_train and y_train again to use on pipeline.fit()??

X=pd.read_csv('data_subset.csv')

X=X.dropna()
Y=X['status'].values
X=X.drop(columns=['status'])

x_train, x_test, y_train, y_test = train_test_split(
    X, Y, stratify=Y, random_state=42, test_size=0.2)


model = lgb.LGBMClassifier(silent=True, subsample=0.8,colsample_bytree=0.2,objective='binary')


parameters={
    'learning_rate_grid_lgbm': [0.015, 0.001, 0.1],    
}

i then run grid search on this data:

clf = GridSearchCV(model, parameters, cv=3, scoring='roc_auc') 
clf.fit(X, y) ### i.e. instead clf.fit(x_train,y_train)

model = clf.best_estimator_

pipeline = make_pipeline(model)

pipeline.fit(x_train, y_train) 
predictions = pipeline.predict_proba(x_test)[:, 1]
roc_auc_score(y_test, predictions)

my confusion is, if I have to use x_train on gridsearch, how should I proceed with the steps after?

Answer 1

Your hyperparameters are chosen based on the whole set of examples, and thus there is a leakage of information of the test set into the model. The validation will, therefore, be too optimistic (as you suspected).

What could be fixed is instead of:

clf.fit(X, y)

just use

clf.fit(x_train, y_train)

As you suggested yourself already. After that line, nothing needs to be changed. You are fitting the model on the training data, and then testing the performance on the (as of yet unseen) test data.

edit: actually, reading the documentation of GridSearchCV it is even a bit worse (depending on how your pipeline.fit method works under the hood): not only are the model hyperparameters chosen on the whole set, by using model = clf.best_estimator_, also your model is now trained using the training + test set, which is a bad thing of course, and will lead to the roc score on the test data to be much higher than on actual unseen data.

Answer 2

If you carry out grid search cross validation on your X data (containing 800k samples), you do not need to make another train_test_split before fitting your model, since the grid search CV strategy already makes several splits (as many as the 'cv' parameter value, check it out here), and then you validate with data never seen before by the model (i.e. the 200k samples you put aside in advance). See the pic below, where your 800k samples are your green training data split into folds, and the left-out 200k samples are your blue test data:

source: https://scikit-learn.org/stable/modules/cross_validation.html

Actually, in your code, you are making that split, but later on you are fitting your model on the whole (X, y) set, not making sense of your train_test split, also calculating your final roc_auc_score with y_test , which in fact is part of the data you used to train (i.e. part of the (X, y) dataset).

Basically, the steps for your example could be:

1. split your dataset to leave 200k out for a final validation as you did
2. apply grid search cross-validation on the rest of the data (800k samples), without splitting it again
3. use clf.best_estimator_ to make predictions on your validation features set to calculate a final ROC AUC

Although not strictly necessary, you can also check the values clf.cv_results_['mean_test_roc_auc'] and clf.cv_results_['std_test_roc_auc'], a couple of arrays where, for each possible hyperparameters combinations (i.e. tried models), you have the mean+std roc_auc value to consider model robustness.