I have a dataset with ~40k records and 16 columns (including the target) and I want to understand the correct process behind whole data science proccess.
This is what I did:
- Performed an EDA which led to me dropping two columns since they were concetrated around one value
- After that I replaced outliers in numerical columns with boundaries based on IQR calculations
- Encoded the target column ( "no" -> 0, "yes" -> 1) using LabelEncoder (per sklearn docs suggestion)
- Encoded the rest of categorical columns with LeaveOneOutEnocoder
- Performed oversampling using SMOTETomek since the target variable value ratio was huge; resulted in additional ~17k records, 56k total
- Split the data using train_test_split
- Modeling (RandomForestClassifier achieved perfect == 1 scores across Accuracy, Precision, Recall and F1Score)
This is where my questions arrise:
- Did I perform data splitting in the right moment? I have some doubts based on the kaggle tutorial on data leakage:
For example, imagine you run preprocessing (like fitting an imputer for missing values) before calling train_test_split(). The end result? Your model may get good validation scores, giving you great confidence in it, but perform poorly when you deploy it to make decisions.
From my understanding the validation data and data used in production is put trough the same preprocessing process as the training data. It's not like we put dirty datasets through live services, sklearn and other libraries would raise multiple execeptions
- Did I perform oversampling in the right moment?
- Is a score of 1 across metrics (on X_test) a good sign? XGBoost was reaching scores of ~0.9, SVM ~0.73, LogisticRegression ~0.8 I do fear that something has gone wrong with that because of the mentioned Kaggle tutorial, altough scores on Kaggle have reached similiar values of ~0.8/0.9
Keep in mind that I'm using data from one CSV file, and operating on two dataframes (X_train, X_test). I understand that there is distinction between validation and test sets.
EDIT: I have performed an experiment, where after loading the data set into a dataframe, I split the csv into two sets (7:3 ratio). I performed all of the above steps on the larger set, yielding similar performance as before. After that I applied all (except SMOTETomek) on the steps to the smaller set (all of the parametrs of encoders, models, outliers etc. remained untouched after initial training on the larger set). The results are, that the models did not drop in performance.