1
$\begingroup$

When training a model we split the dataset into training set and test set. In case a normalization/standardization is needed on any column then this process is done separately for training set and test set to prevent data leakage. The normalization of the test set is done on the parameters of training set. The code that we usually write is something like below -

scaler = StandardScaler() #scaler = MinMaxScaler()
train = scaler.fit_transform(train)
test = scaler.transform(test)

The fit part figures out the parameters and then transform part does the normalization.

Now my question is - When the model is deployed in production, the column from real life data also needs to be normalized. How do we do that? Do we store this scaler somewhere (may be pickle) or do we store the parameters of the scaler (like min, max, mean, standard deviation etc.) somewhere (e.g. a file, table) and then read the stored file or table to retrieve the values of parameters and perform normalization of the new real life data before calling the predict function.

$\endgroup$
0
$\begingroup$

It depends on the max value of each column: If the real life data has new max values, the previous scaler could be wrong and you may have to rescale it. Therefore, you can test if the scaler is correct.

To save your scaler, you can use the dump function from pickle:

from pickle import dump
dump(scaler, open('scaler.pkl', 'wb'))

Then you can load it:

from pickle import load
scaler = load(open('scaler.pkl', 'rb'))

To test it, you can check the min and max values described here: https://machinelearningmastery.com/how-to-save-and-load-models-and-data-preparation-in-scikit-learn-for-later-use/

$\endgroup$
1
  • $\begingroup$ If you consider the answers somewhat usefull, don't hesitate to upvote them as acknowledgment :) $\endgroup$ Jul 22 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.