Normalization of production data

Question

When training a model we split the dataset into training set and test set. In case a normalization/standardization is needed on any column then this process is done separately for training set and test set to prevent data leakage. The normalization of the test set is done on the parameters of training set. The code that we usually write is something like below -

scaler = StandardScaler() #scaler = MinMaxScaler()
train = scaler.fit_transform(train)
test = scaler.transform(test)

The fit part figures out the parameters and then transform part does the normalization.

Now my question is - When the model is deployed in production, the column from real life data also needs to be normalized. How do we do that? Do we store this scaler somewhere (may be pickle) or do we store the parameters of the scaler (like min, max, mean, standard deviation etc.) somewhere (e.g. a file, table) and then read the stored file or table to retrieve the values of parameters and perform normalization of the new real life data before calling the predict function.

Answer 1

It depends on the max value of each column: If the real life data has new max values, the previous scaler could be wrong and you may have to rescale it. Therefore, you can test if the scaler is correct.

To save your scaler, you can use the dump function from pickle:

from pickle import dump
dump(scaler, open('scaler.pkl', 'wb'))

Then you can load it:

from pickle import load
scaler = load(open('scaler.pkl', 'rb'))

To test it, you can check the min and max values described here: https://machinelearningmastery.com/how-to-save-and-load-models-and-data-preparation-in-scikit-learn-for-later-use/