1
$\begingroup$

I was looking at the source codes of MinMaxScaler on Github. I know that when you fit a preprocessing class to a dataset, it takes the data and prepares it for transformation.

Let's say, I fitted MinMaxScaler to X_train and transformed it. But how does transform work when I use another dataset, let's say X_test? When you call transform(), does it replace the datasets in the use?

$\endgroup$
1
$\begingroup$

Models are trained on training data, and evaluated on test data based on assumption that unseen test data also comes from the same distribution with training data. So when you calculated statistics for training data, based on assumption that test data also comes from the same distribution you should apply same transformations to test data. You should fit MinMaxScaler to your training data, and then use this scaler to transform both tranining data and test data. There are also issues about data leakage, take a look at that: StandardScaler before and after splitting data

For transformations, fit method extract relevant statistics(min, max value for min-max scaling, mean, std for standardization) from the provided data, and transform method transforms each feature individually based on extracted statistics.

$\endgroup$
2
  • $\begingroup$ Thanks for the answer, but I'm interested in how the code works. For example, the fit method does nothing other than returning partial_fit. To be precise, it first resets the internal state and then returns partial_fit. If transform uses fit for transforming, why doesn't it take partial_fit as input? $\endgroup$
    – Rutrasss
    May 4 at 21:25
  • $\begingroup$ MinMaxScaler is a class right? So it has some attributes to do scaling like data_min_, data_max_, scale_. partial_fit is for online fitting, for cases like data is coming from stream, and you want to extract statistics in an online manner. Nevertheless it can be used for batch case too, and it is what fit function does. Instead of repeating same code again it just makes a call to partial_fit to calculate statistics, and store them in attributes. Then what transform does is only making transform by using attributes of the class. $\endgroup$
    – tkarahan
    May 5 at 4:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.