2
$\begingroup$

I am new to machine learning. I am bit confused in preprocessing. Generally,

Scenario-1: I am splitting the dataset into train,test and validation and applying the transformations like fit_transform on train and transform on test.

Scenario-2: The other method is applying transformations on the entire dataset first and then split the dataset into train,test and validation. I am bit confused in choosing , dividing the data before preprocessing and feature engineering or after preprocessing and feature engineering. Looking for a nice answer with effects and casues.

$\endgroup$
  • 5
    $\begingroup$ Hi - you want to fit the preprocessing transformations on your training set, and then apply that transformation to all of your data. This way you're not leaking information about the distribution of your test and validation sets into the transformation you're applying to your training data and whatever approach you take downstream. More details and link to crossvalidated discussion: datascience.stackexchange.com/questions/38395/… $\endgroup$ – redhqs Mar 7 '19 at 9:57
  • $\begingroup$ You will first need to preprocess your dataset and then split it. $\endgroup$ – Shubham Panchal Mar 7 '19 at 10:37
1
$\begingroup$

You should absolutely adopt the first scenario. That's because the transformers that you use have some parameters (e.g. mean and standard deviation in case of standard scalar) and this parameters are learned from data like the parameters of your machine learning model. As you know, you should not use the validation and test data for learning the model parameters and for the same reason, you should not use them for learning the transformer parameters. As a result, you should just use the training samples for fitting your transformer parameters if you want to try a practical machine learning scenario.

$\endgroup$
-2
$\begingroup$

The prepossessing transformations have to be applied on all data sets ( train, test and validation). Whatever feature you add on one subset, has to be added in the other subsets.

If you're validation and test data are chosen randomly and you apply the exact transformations on all subsets both scenarios would work but the second scenario is better in some ways:

  • First it easier to code you'll just have to write the transformation lines once.
  • Second it is probably faster, you'll benefit from the power of numpy if you're using python for example
  • Third it is less risky, take the following example. You have a categorical variable and for some reason you decide to replace the missing values in that variable with the value with highest occurrence. If the value occurrences are close you could have different value in your subsets. The same could happen if you replace a numerical value with the mean. This is unlikely if you pick your subsets randomly but you know, there is no prefect randomness.

I would recommend choosing the second option as it is easier and less risky. But still, you can try both at some point and you'll see the advantages and disadvantages of each.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.