What is the right processing order when working with a dataset that already consists of test and train data?

Question

I want to work on the following task:

• Text Classification using Deep Learning models and a Transfer Learning model.
• The notebook that I'm creating should include the following steps:
  1. Data Preparation
  2. Cleaning
  3. EDA
  4. Embedding/Preprocessing
  5. Training
  6. Evaluation
• The data set that I want to use is the yahoo_answer_topics from Hugging Face and comes with an already defined train and test split. And here comes the point I'm not sure about: How should I deal with this predefined train and test split?

Consulting other similar questions about when to do the train and test split, it is stated that the cleaning and EDA can be performed on the entire dataset. However, other resources mention that the EDA should only be done on the training data and use the validation and test data to evaluate the quality of any decisions made on the train data set. What is considered the correct way?

If the first would be the way to go; would this mean I have to best combine the two datasets at the start, clean, preprocess, and do the EDA before splitting them again into test, validation, and test?

If the second would be the way to go; would this mean I have to split the train into validation and train as a first step and then do the cleaning, and EDA only on the train data set, and then apply the same cleaning/transformation steps on the validation and test data right before training the model? Would it, therefore, be a violation if I look at the validation and test data set before the training (e.g. checking the distribution of the different classes in the validation and test set) and what if I want to do K fold Cross Validation?

Thank you already for your help!

Answer 1

In my opinion, it's simpler to always assume that the model is meant to be applied in the future on some unknown data:

• In theory, this is always the goal of training a supervised ML model. Even if this is done only for educational purposes, it makes sense to practice with realistic constraints.
• Practically, this requires to design the system in a way such that any required preprocessing can be re-run with new data. Doing things this way prevents any risk of data leakage and ensures that we properly evaluate the test set.

Of course it's common to apply some steps on the whole dataset to save time, but the proper way is always to split first (if needed) and work only with the training set until the step of evaluation, considering the test set like as unknown future data.

• The preprocessing steps should be coded in a function which can be applied to any subset of data.
• There's no need to study the distribution of the evaluation or test set:
  • in general, these sets should have the same distribution as the training set, so they don't bring any new information.
  • occasionally the test set is purposefully built with a slightly different distribution because this corresponds to a realistic application: in this case it would clearly be a mistake to use this information before training (data leakage), since the goal is to evaluate the robustness of the model.
• K-fold cross validation would be a different setting, since it relies on evaluating with different subsets of data. For example you could use CV on the training set in order to tune some hyper-parameters, and later evaluate the final model with its best parameters on the test set.