I want to work on the following task:
- Text Classification using Deep Learning models and a Transfer Learning model.
- The notebook that I'm creating should include the following steps:
- Data Preparation
- Cleaning
- EDA
- Embedding/Preprocessing
- Training
- Evaluation
- The data set that I want to use is the yahoo_answer_topics from Hugging Face and comes with an already defined train and test split. And here comes the point I'm not sure about: How should I deal with this predefined train and test split?
Consulting other similar questions about when to do the train and test split, it is stated that the cleaning and EDA can be performed on the entire dataset. However, other resources mention that the EDA should only be done on the training data and use the validation and test data to evaluate the quality of any decisions made on the train data set. What is considered the correct way?
If the first would be the way to go; would this mean I have to best combine the two datasets at the start, clean, preprocess, and do the EDA before splitting them again into test, validation, and test?
If the second would be the way to go; would this mean I have to split the train into validation and train as a first step and then do the cleaning, and EDA only on the train data set, and then apply the same cleaning/transformation steps on the validation and test data right before training the model? Would it, therefore, be a violation if I look at the validation and test data set before the training (e.g. checking the distribution of the different classes in the validation and test set) and what if I want to do K fold Cross Validation?
Thank you already for your help!