3
$\begingroup$

I was reading a lot recently about PCA and cross validation and it seems that the majority call it malpractice to do PCA before cross validation. I would also like to perform SMOTE, but there is a split between those who perform SMOTE before or after PCA. Yes, i am very confused, but even authors of published papers seem to be confused as well about this (Me being a newbie). Even here (on CrossValidate) there are different answers for the same Q about SMOTE and PCA, and the gist is, see what suits your data. Anyway, I know how to perform each by it's own, but don't know how to do PCA in each fold, for example in Kfold, or any other cross validation method.

1- Is it possible for someone to share with me how to do PCA inside Kfold? Through a loop or pipeline maybe?

2- I personally have a bipolar opinion on the SMOTE/PCA thing: a- I think that PCA should be performed before SMOTE so that the dimentionality reduction is done on the main (real) training data before we sample it (In my case upsampling). b- My other thought on it, is that if you upsample and then perform SMOTE then your dimentionality reduction is based on "non-biased" group sample (where both groups now have same number). My Q here is, if i CV (cross validated) PCA, can i include SMOTE in the mix?

3- Last Q, I saw a video of a prof. who said that if you test/split your data and then CV PCA on the training set, this is wrong. Since cross validate e.g. Kfold already does the train/test split for you. Is this the case? If so, then how would i eventually select my X_test X_train y_test y_train for further analysis?

$\endgroup$
3
$\begingroup$

To avoid data leakage, it's important to do any transformations after you split your data - so yes, you want to fit PCA to your training set/fold, but apply that transformation to both training and test data (when you want to consider generalisation).

Scikit-learn makes this process straightforward with sklearn.pipeline.Pipeline and there's a great example where 5-fold CV is done on a chain of PCA to logistic regression which should give you something build on.

To address your second question, there's nothing stopping you from including SMOTE in the mix after doing your train-test or CV split, but without knowing more about the data and situation I won't comment further.

Finally, it is not necessary to nest a train-test split within k-fold CV, since as you say, k-fold CV incorporates a test set. However, for final model selection, a more rigorous approach would be to split off a final validation set before you do anything else. Carry out your k-fold CV experiments on that training data to choose whatever model and parameters, and use the validation results as a final estimate of generalisation.

To implement this, the scikit-learn example above shows how you can evaluate the effect of different parameters on the classifier. What it does not show is final model selection and evaluation, but all that needs to be done is split off a reasonable amount of data prior to everything in the example, and then take the best model(s) and evaluate their performance on that validation set.

$\endgroup$
  • $\begingroup$ Concerning Q1 Great example, thanks! So i just pipe = Pipeline(steps=[('pca', pca), (‘svc’, svc)]) and then enter the pipeline into GS like search = GridSearchCV(pipe, param_grid, iid=False, cv=5, return_train_score=False) search.fit(X_digits, y_digits) that should work right?, if i already classified what the use of SMOTE AFTER? Q2- A long discussion about this in the community, so ill just see others opinions about this. Q3- So you are suggesting that i train/test split in the very beginning and then do Kfold on the train set only and use the untouched test set at the very end to check? $\endgroup$ – tsumaranaina Feb 22 at 6:57
  • 1
    $\begingroup$ That pipeline should work! As long as param_grid covers what you'd like it to do and given you're calling pca and svc correctly. Sorry if I wasn't clearer, I meant to say [smote -> pca -> svc] or [pca -> smote - svc], as you say there's no point in smote post-svc. On Q3 yes exactly. Some might say it's overkill but I think it's the closest you can get to unseen data. Last thing don't forget to stratify your splits, especially considering imbalanced classes. Back to Q2 if you have time, try both (I have opinions but no useful references) $\endgroup$ – redhqs Feb 22 at 10:18
  • 1
    $\begingroup$ Thanks again. I'll let you know what i find out! $\endgroup$ – tsumaranaina Feb 22 at 10:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.