How do I use the same scale used in preprocessing with new data.

Actual code:

x = df.values #returns a numpy array
min_max_scaler = preprocessing.MinMaxScaler()
x_scaled = min_max_scaler.fit_transform(x)
df_scaled = pd.DataFrame(x_scaled)

clf = tree.DecisionTreeClassifier()
clf.fit(X_train, y_train)
pred = clf.predict(X_test)

If I understand it correctly I should have included a scaler variable with the StandardScaler.


Something like:

clf = tree.DecisionTreeClassifier()
clf.fit(X_train, y_train)
scaler = preprocessing.StandardScaler().fit(X_train)
pred = clf.predict(X_test)

What scaler parameters should I use for future data processing?



IMO, you don't need to use scaling if your classifiers are based on decision trees. Also, in your final piece of code, the variable scaler is never used, so I am not sure at all why it is defined. Nevertheless, if you insist on using a scaler, you should Pipeline it so it automatically applies to the test data the same scaling it has learned by fitting to training data:

pipeline = Pipeline([('scaler', StandardScaler()), ('classifier', DecisionTreeClassifier())])
pipeline.fit(X_train, y_train)
predictions = pipeline.predict(X_test)

Note that you'd obviously need to import the necessary modules (I was too lazy to write them here!).

  • $\begingroup$ I edited the question adding the scaling code I used. I am actually running several algos to compare performance. Some need data scaling. $\endgroup$ – Diego Oct 12 '17 at 20:46
  • $\begingroup$ In your edits, df_scaled is not used elsewjere, and you are still training on X_train and y_train which are derived from df (prior to scaling). Anyhow, you can use the Pipeline boilerplate code in my answer with all your classifiers; just replace DecisionTreeClassifier() with your classifier of choice. $\endgroup$ – darXider Oct 12 '17 at 21:04
  • $\begingroup$ Thanks!, if I used the min_max_scaler as per my edition, should I just replace StandardScaler in the Pipeline code? $\endgroup$ – Diego Oct 12 '17 at 21:09
  • $\begingroup$ Yes, just use MinMaxScaler() instead of StandardScaler(). $\endgroup$ – darXider Oct 13 '17 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.