I've been given 2 datasets , and there are missing values in both the test and training data set. Do I need to pre-process test.csv also or is it only for train.csv?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ yes. fit on train, transform on train and test $\endgroup$– Neil McGuiganCommented Oct 17, 2021 at 18:30
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Preprocessing is needed for both train and test sets. But you should be aware of data leakage, meaning no information from the test set should be used to preprocess the training set.
For example, if you are trying to apply One-Hot encoding to your classification labels you should train the encoder (e.g. sklearn.preprocessing.OneHotEncoder) on training set and apply the trained encoder to get the labels for test set.
Or if you want to normalize a feature, calculate the mean and standard deviation from the training set and use it to normalize both training and test sets.
$\begingroup$
$\endgroup$
The main reason for data preprocessing is to ensure that the datasets are formatted in such a way that the data they contain can be interpreted and parsed by the machine learning algorithms.
As you train the machine learning models with training data and predict with machine learning models with the test data. Data preprocessing has to be applied to both train and test data