Order of preproccesing, avoiding leakage and metrics

Question

I have a dataset with ~40k records and 16 columns (including the target) and I want to understand the correct process behind whole data science proccess.

This is what I did:

1. Performed an EDA which led to me dropping two columns since they were concetrated around one value
2. After that I replaced outliers in numerical columns with boundaries based on IQR calculations
3. Encoded the target column ( "no" -> 0, "yes" -> 1) using LabelEncoder (per sklearn docs suggestion)
4. Encoded the rest of categorical columns with LeaveOneOutEnocoder
5. Performed oversampling using SMOTETomek since the target variable value ratio was huge; resulted in additional ~17k records, 56k total
6. Split the data using train_test_split
7. Modeling (RandomForestClassifier achieved perfect == 1 scores across Accuracy, Precision, Recall and F1Score)

This is where my questions arrise:

1. Did I perform data splitting in the right moment? I have some doubts based on the kaggle tutorial on data leakage:

For example, imagine you run preprocessing (like fitting an imputer for missing values) before calling train_test_split(). The end result? Your model may get good validation scores, giving you great confidence in it, but perform poorly when you deploy it to make decisions.

From my understanding the validation data and data used in production is put trough the same preprocessing process as the training data. It's not like we put dirty datasets through live services, sklearn and other libraries would raise multiple execeptions

2. Did I perform oversampling in the right moment?
3. Is a score of 1 across metrics (on X_test) a good sign? XGBoost was reaching scores of ~0.9, SVM ~0.73, LogisticRegression ~0.8 I do fear that something has gone wrong with that because of the mentioned Kaggle tutorial, altough scores on Kaggle have reached similiar values of ~0.8/0.9

Keep in mind that I'm using data from one CSV file, and operating on two dataframes (X_train, X_test). I understand that there is distinction between validation and test sets.

EDIT: I have performed an experiment, where after loading the data set into a dataframe, I split the csv into two sets (7:3 ratio). I performed all of the above steps on the larger set, yielding similar performance as before. After that I applied all (except SMOTETomek) on the steps to the smaller set (all of the parametrs of encoders, models, outliers etc. remained untouched after initial training on the larger set). The results are, that the models did not drop in performance.

Answer 1

The split should be performed at the beggining if you want to avoid any form of leakage. As is you don't have that many leaky step, except SMOTE and maybe the IQR clipping.

Other remarks:

• 1. removing feature based only on EDA is tricky, as is you could be removing important info.
• 2. replacing outliers is not necessary if you are going to use tree models.
• 3. & 4) your approach seems correct
• 5. the first approach to dealing with evantual imbalance is to do nothing and see if there is any problem. SMOTE might be very tricky to use esp. for explaining what your model do. Try without it first.
• 6. generally perform the split as early as possible.
• 7. perfect score might indicate some problem. Check what score other people get. If the score hold it usually mean that there is a simpler model lying somewhere, try to find it.

Also generally speaking you might want to share your notebook (here and on kaggle) and ask on kaggle.