Question about grid search and KFold

Question

I am trying an example which I am training on a huge dataset 5M (only 4 features) rows with Cudf and CUml and I am using SGD logistic regression because I must predict if the patient if is sick or not .

I am using stratify kfold because the dataset is has like 20474 infected and the rest 4_979_256 are healthy. I will be using as a metric the Recall or f1 score.

I have done 5 stratify kfold and now I want to do a grid search.

I have seen on the manual https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html

(cuml has the same parameters)

My question s: cv = 5 (default) I want to ask if I do 5 cv I must 5 DIFFERENT One hot encodings and normalizations (or else I will have a data leakage) how u tackle this u just run once the grid search without kfold just a split train and test with a specific random state and afterwards u run 5,10,etc cv and if the scores are okay u say okay finish? or I am missing something

Answer 1

Start by splitting the data into train and test data. The test set should remain unseen until you do your final evaluation of the model on it. It should also not be included when fitting preprocessing functions like normalization, scaling, etc. to prevent data leakage.

Then, with the train data you run the GridSearchCV, which will automatically split the train data into 5 parts (if cv = 5) and each of these parts will once be a validation set for the grid search and the rest of the time be a training set for the grid search.

Finally, when GridSearchCV is done, it will fit a model with the best found model parameters on the entire training set that has been passed to the GridSearchCV function. This model should then be the best performing one and can be used for evaluation on the unseen test set.