I am new to Machine Learning and I have a conceptual question.

I have a scaled dataset (scikit-learn and pandas).

After training/testing my algo, I will make new predictions using new actual data which will not be scaled or normalized.

Will this discrepancy be a problem, if so, how should I resolve it?


  • 1
    $\begingroup$ what do you mean by scaled and normalized? Could you explain more in-depth relating to your data? $\endgroup$ Oct 2 '17 at 18:11
  • $\begingroup$ That does sound like trouble from the start. Usually, the training data distribution should match that of the data you wish to predict. We could use more details about your data set. $\endgroup$
    – E_net4
    Oct 2 '17 at 19:22
  • $\begingroup$ @RahulAedula I am using from sklearn import preprocessing x = result.values #returns a numpy array min_max_scaler = preprocessing.MinMaxScaler() x_scaled = min_max_scaler.fit_transform(x) result_scaled = pd.DataFrame(x_scaled) $\endgroup$
    – Diego
    Oct 2 '17 at 20:04
  • $\begingroup$ @E-net4, from what I learned the suggestion is to scale the data before training the algo (especially when the features have very different scales, ie stock prices, volume, return %s). My dataset is very extensive $\endgroup$
    – Diego
    Oct 2 '17 at 20:06

You should save the scaler params used to fit the training set and use the same ones to transform all other data used with the model from then on - whether CV, test or new unseen data.

After training/testing my algo, I will make new predictions using new actual data which will not be scaled or normalized.

No that won't work. Once you add scaling/normalisation to the training pipeline, the exact same scaling (as in same scaling params, not re-calculated) should be applied to all input features.

The scikit-learn scalers like e.g. StandardScaler have two key methods:

  • fit should be applied to your training data

  • transform should be applied after fit, and should be used on every data set to normalise model inputs.

fit_transform can be used on the training data only to do both in a single step.

If you need to do the training and predictions in different processes (maybe live predictions are on different devices for instance), then you need to save and restore the scaling params. One basic, simple way to do this is using pickle e.g. pickle.dump( min_max_scaler, open( "scaler.p", "wb" ) ) to save to a file and min_max_scaler = pickle.load( open( "scaler.p", "rb" ) ) to load it back.

  • $\begingroup$ Thank you for clearifying this. The scaler params are set by scikit learn, how do I get them and apply them to the actual data I want to make predictions with? I am not talking about the testing portion of the process but the real prediction. $\endgroup$
    – Diego
    Oct 2 '17 at 20:45
  • $\begingroup$ I found this but do not completely understand how to implement it: stackoverflow.com/questions/35944783/… $\endgroup$
    – Diego
    Oct 2 '17 at 20:47
  • $\begingroup$ Thanks a lot!! One aditional issue, I built my dataset from different sources and I preprocessed each one separately (probably not a good idea). If I understand it correctly then I will have different scaler params. Should I build my entire dataset first and at the end perform the preprocess? $\endgroup$
    – Diego
    Oct 2 '17 at 20:52
  • $\begingroup$ @Diego: Yes if you have built and scaled the data piecemeal that might not work very well (although if you dataset is very large then the differences might be small enough it will not matter very much). Best to have one trained scaler and re-use it. The scaling doesn't usually have to be super-precise, so if you have millions of records, you can build the scaler from one large enough data set, save it and then apply it to everything else. However yes if it is simple to combine first then scale, do that . . . $\endgroup$ Oct 2 '17 at 21:09
  • $\begingroup$ I appreciate so much your follow up. Thank you very much !!! $\endgroup$
    – Diego
    Oct 2 '17 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.