Separating the features data from the target in X and y before or after a pipeline?

Question

I have the following:

train_set, test_set = train_test_split(arbres_df, test_size=0.2, random_state=42)

Which is the old train_test_split we know.

And then I separate the features and the target:

train_feat = train_set.drop("anneedeplantation", axis=1).reset_index(drop=True)
train_target = train_set["anneedeplantation"].copy().reset_index(drop=True)

Following through the famous book Hands-on Machine Learning I have a little issue with building a pipeline:

The author uses train_test_split in the beginning of the project, he also separates the target and the feature in X_train and y_train do some analysis in the train_data, build some new features etc and then put everything in a pipeline to transform the data. Everything is fine.

Now I am working on another project and one thing came out different: after splitting the data I needed to drop some rows in the train_feat. In the book he didn't had to drop any rows, that is why when transforming the data in the pipeline gives no problem. But I needed to drop some rows and I ended up with a train_feat with less rows than the train_target.

My questions are:

• How can I insert in the pipeline the instruction to drop the equivalent rows in the train_target? I can't drop any row it must be the rows that corresponds to the ones that were dropped in the train_features.

• Is the answer to the above question simply: "don't separate the data in X_train and y_train before the pipeline but only after"? If so, is this good practice?

Answer 1

To answer your questions, you do train-test-split before you fit the pipeline. You don't need to do the row-dropping in the pipeline; you do this before you do the train-test-split. In other words, you drop the rows you want to drop before splitting the dataset, so then there will be no issue of dropping the rows in the train set but the rows still being there in the test set.

It doesn't really matter whether you do train-test-split before or after you "build" the pipeline, but it does matter that you do it before you fit the pipeline on the data. If you fit it on the whole dataset, you don't have anywhere to test it anymore!

Hope this helps!