2
$\begingroup$

Let's say I used minmaxscaler while creating my model. Now, i'm loading that model via Pickle in a Flask app. Upon receiving a request containing a datapoint I would like to apply to it the same transformations that I applied to my training dataset before calling the predict() method. How do I transfer that set of transformations from one file to a webservice?

$\endgroup$
2
$\begingroup$

Rather than storing and loading many files, create a Scikit-learn transformation pipeline with all of your transformations, and then save that as a pickle or joblib file.

from sklearn.pipeline import Pipeline
from sklearn.externals import joblib

pipeline = Pipeline([
                ('normalization', MinMaxScaler()),
                ('classifier', RandomForestClassifier())
            ])

joblib.dump(pipeline, 'transform_predict.joblib')

You can then just load one transformation pipeline and call fit_transform to transform the input data and get predictions for it:

 pipeline = load('transform_predict.joblib') 
 predictions = pipeline.predict(new_data)
$\endgroup$
  • 1
    $\begingroup$ Thanks, this is what i was looking for $\endgroup$ – Blenzus Mar 26 at 14:43
  • $\begingroup$ Does this apply to dummy variables? $\endgroup$ – Blenzus Mar 26 at 14:55
  • 1
    $\begingroup$ If you're using scikit-learn's OneHotEncoder then yes. Any scikit learn 'transformer' can be used with a pipeline, so anything that implements the TransformerMixin and BaseEstimator: github.com/scikit-learn/scikit-learn/blob/7b136e9/sklearn/… This also means you can create your own custom 'transformers' to add to a pipeline, by implementing these in the same way. $\endgroup$ – Dan Carter Mar 26 at 15:34
1
$\begingroup$

You need to save minmaxscaler (along with model). In Flask app, you can :

  1. Load scaler from file
  2. Use this instance of scaler for scaling input values

#While training

from sklearn.externals import joblib
scaler_filename = "saved_scaler"
joblib.dump(scaler, scaler_filename)

In Flask App

scaler_filename = "saved_scaler"    
scaler = joblib.load(scaler_filename)
$\endgroup$
  • $\begingroup$ Do i need to do this for every normalization library i use? i'll be loading many files into the memory, isn't there way to load something that contains every step of the data transformations? $\endgroup$ – Blenzus Mar 26 at 14:17
  • 1
    $\begingroup$ You can save and load all scalers at the same time. Example : stackoverflow.com/questions/33497314/… $\endgroup$ – Shamit Verma Mar 26 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.