I'm using decision tree classification for a classification problem. I have preprocessed the data, train/test split it, and run a model with cross validation before testing it. The steps I followed for preprocessing are outlined below:
- Removed some occurences (rows) which aren't usable
- Transformed some of the columns by taking nth-root to remove skew (n is different for each column, I plotted the data and did whatever looked like it reduced the skew most)
- Train/test split the data
- I fit
StandardScaler()to the training data
- I applied the transformations in step 4 to both the training and test data
My questons are as follows:
- Are my steps correct? In particular, is it correct to 'root transform' the data before train/test split, or does that lead to data leakage?
- When I want to apply my model to new data (after testing etc.) does that new data have to undergo identical preprocessing? e.g. fit to the train set then apply it to the new data and root transformations of the same nth-root.
Thanks in advance