1
$\begingroup$

I am wondering why do we use scaling on train and test set separately. I understand that transform () on test data μ and σ as computed from fit_transform() on Train. But why can we compute μ and σ from all given data (before split) and then apply them on future data.

Do we do this because we don't know how the size of our future data?

$\endgroup$
0
0
$\begingroup$

You must use the same scaling factors (μ and σ) for training and test data. The reason that you must use the same scaling factors is that you should compare the data with the same pre-processing. Note that scaling factors (μ and σ) must be computed using only the training data. That is because in practice you are not provided with test data and you just have to evaluate your model on test data.

$\endgroup$
1
  • $\begingroup$ Yes, I got it. We want our parameters to accurately (or maybe closely) predict the y_test before they predict the future data sets. $\endgroup$ Aug 9 '18 at 16:38
-2
$\begingroup$

train and test datasets should have no overlap of data ... so when you scale each they may very well have different scaling factors

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.