# How to correctly apply the same data transformation , used on the training dataset , on real data in a webservice?

Let's say I used minmaxscaler while creating my model. Now, i'm loading that model via Pickle in a Flask app. Upon receiving a request containing a datapoint I would like to apply to it the same transformations that I applied to my training dataset before calling the predict() method. How do I transfer that set of transformations from one file to a webservice?

Rather than storing and loading many files, create a Scikit-learn transformation pipeline with all of your transformations, and then save that as a pickle or joblib file.

from sklearn.pipeline import Pipeline
from sklearn.externals import joblib

pipeline = Pipeline([
('normalization', MinMaxScaler()),
('classifier', RandomForestClassifier())
])

joblib.dump(pipeline, 'transform_predict.joblib')


You can then just load one transformation pipeline and call fit_transform to transform the input data and get predictions for it:

 pipeline = load('transform_predict.joblib')
predictions = pipeline.predict(new_data)

• Does this apply to dummy variables? – Blenz Mar 26 at 14:55
• If you're using scikit-learn's OneHotEncoder then yes. Any scikit learn 'transformer' can be used with a pipeline, so anything that implements the TransformerMixin and BaseEstimator: github.com/scikit-learn/scikit-learn/blob/7b136e9/sklearn/… This also means you can create your own custom 'transformers' to add to a pipeline, by implementing these in the same way. – Dan Carter Mar 26 at 15:34

You need to save minmaxscaler (along with model). In Flask app, you can :

2. Use this instance of scaler for scaling input values

#While training

from sklearn.externals import joblib
scaler_filename = "saved_scaler"
joblib.dump(scaler, scaler_filename)


scaler_filename = "saved_scaler"

• Do i need to do this for every normalization library i use? i'll be loading many files into the memory, isn't there way to load something that contains every step of the data transformations? – Blenz Mar 26 at 14:17
• You can save and load all scalers at the same time. Example : stackoverflow.com/questions/33497314/… – Shamit Verma Mar 26 at 14:33

I just noticed that none of the answers contained the most important instruction. So here is how it's done since i already resolved this a long time ago.

from sklearn.pipeline import Pipeline
from sklearn.externals import joblib

pipeline = Pipeline([
('normalization', MinMaxScaler()),
('classifier', RandomForestClassifier())
])
pipeline.fit(X)


The pipeline.fit() was missing from other answers which is very important. Once scikit function has learned the parameters of your training data X , you can dump that pipeline into a file and do the rest like this.

joblib.dump(pipeline, 'transform_predict.joblib')