PCA, SMOTE and cross validation- how to combine them together?

Question

I was reading a lot recently about PCA and cross validation and it seems that the majority call it malpractice to do PCA before cross validation. I would also like to perform SMOTE, but there is a split between those who perform SMOTE before or after PCA. Yes, i am very confused, but even authors of published papers seem to be confused as well about this (Me being a newbie). Even here (on CrossValidate) there are different answers for the same Q about SMOTE and PCA, and the gist is, see what suits your data. Anyway, I know how to perform each by it's own, but don't know how to do PCA in each fold, for example in Kfold, or any other cross validation method.

1- Is it possible for someone to share with me how to do PCA inside Kfold? Through a loop or pipeline maybe?

2- I personally have a bipolar opinion on the SMOTE/PCA thing: a- I think that PCA should be performed before SMOTE so that the dimentionality reduction is done on the main (real) training data before we sample it (In my case upsampling). b- My other thought on it, is that if you upsample and then perform SMOTE then your dimentionality reduction is based on "non-biased" group sample (where both groups now have same number). My Q here is, if i CV (cross validated) PCA, can i include SMOTE in the mix?

3- Last Q, I saw a video of a prof. who said that if you test/split your data and then CV PCA on the training set, this is wrong. Since cross validate e.g. Kfold already does the train/test split for you. Is this the case? If so, then how would i eventually select my X_test X_train y_test y_train for further analysis?

Answer 1

To avoid data leakage, it's important to do any transformations after you split your data - so yes, you want to fit PCA to your training set/fold, but apply that transformation to both training and test data (when you want to consider generalisation).

Scikit-learn makes this process straightforward with sklearn.pipeline.Pipeline and there's a great example where 5-fold CV is done on a chain of PCA to logistic regression which should give you something build on.

To address your second question, there's nothing stopping you from including SMOTE in the mix after doing your train-test or CV split, but without knowing more about the data and situation I won't comment further.

Finally, it is not necessary to nest a train-test split within k-fold CV, since as you say, k-fold CV incorporates a test set. However, for final model selection, a more rigorous approach would be to split off a final validation set before you do anything else. Carry out your k-fold CV experiments on that training data to choose whatever model and parameters, and use the validation results as a final estimate of generalisation.

To implement this, the scikit-learn example above shows how you can evaluate the effect of different parameters on the classifier. What it does not show is final model selection and evaluation, but all that needs to be done is split off a reasonable amount of data prior to everything in the example, and then take the best model(s) and evaluate their performance on that validation set.