4
$\begingroup$

Let's say I used minmaxscaler while creating my model. Now, i'm loading that model via Pickle in a Flask app. Upon receiving a request containing a datapoint I would like to apply to it the same transformations that I applied to my training dataset before calling the predict() method. How do I transfer that set of transformations from one file to a webservice?

$\endgroup$

3 Answers 3

7
$\begingroup$

I just noticed that none of the answers contained the most important instruction. So here is how it's done since i already resolved this a long time ago.

from sklearn.pipeline import Pipeline
from sklearn.externals import joblib

pipeline = Pipeline([
                ('normalization', MinMaxScaler()),
                ('classifier', RandomForestClassifier())
            ])
pipeline.fit(X)

The pipeline.fit() was missing from other answers which is very important. Once scikit function has learned the parameters of your training data X , you can dump that pipeline into a file and do the rest like this.

joblib.dump(pipeline, 'transform_predict.joblib')
pipeline = load('transform_predict.joblib') 
transformed_data = pipeline.transform(new_data)

This will apply the same transformations to a dataset (even if containing 1 sample) that was done on your training dataset.

Update

for newer version, I believe one has to read. I hit the memory limit and the following change helped.

with open(conf['le_embedder_file'], 'rb') as f:
    pipe_le = joblib.load(f)  

and write

with open(conf['tecv_embedder_file'], 'wb') as f:
    joblib.dump(pipe_tecv, f, compress='zlib')

I had to read

$\endgroup$
3
$\begingroup$

Rather than storing and loading many files, create a Scikit-learn transformation pipeline with all of your transformations, and then save that as a pickle or joblib file.

from sklearn.pipeline import Pipeline
from sklearn.externals import joblib

pipeline = Pipeline([
                ('normalization', MinMaxScaler()),
                ('classifier', RandomForestClassifier())
            ])

joblib.dump(pipeline, 'transform_predict.joblib')

You can then just load one transformation pipeline and call fit_transform to transform the input data and get predictions for it:

 pipeline = load('transform_predict.joblib') 
 predictions = pipeline.predict(new_data)
$\endgroup$
2
  • $\begingroup$ Does this apply to dummy variables? $\endgroup$
    – Blenz
    Mar 26, 2019 at 14:55
  • 1
    $\begingroup$ If you're using scikit-learn's OneHotEncoder then yes. Any scikit learn 'transformer' can be used with a pipeline, so anything that implements the TransformerMixin and BaseEstimator: github.com/scikit-learn/scikit-learn/blob/7b136e9/sklearn/… This also means you can create your own custom 'transformers' to add to a pipeline, by implementing these in the same way. $\endgroup$
    – Dan Carter
    Mar 26, 2019 at 15:34
1
$\begingroup$

You need to save minmaxscaler (along with model). In Flask app, you can :

  1. Load scaler from file
  2. Use this instance of scaler for scaling input values

#While training

from sklearn.externals import joblib
scaler_filename = "saved_scaler"
joblib.dump(scaler, scaler_filename)

In Flask App

scaler_filename = "saved_scaler"    
scaler = joblib.load(scaler_filename)
$\endgroup$
2
  • $\begingroup$ Do i need to do this for every normalization library i use? i'll be loading many files into the memory, isn't there way to load something that contains every step of the data transformations? $\endgroup$
    – Blenz
    Mar 26, 2019 at 14:17
  • 1
    $\begingroup$ You can save and load all scalers at the same time. Example : stackoverflow.com/questions/33497314/… $\endgroup$ Mar 26, 2019 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.