1
$\begingroup$

I do understand the concept of normalizing & scaling the training/test data; it does help with the converging of the cost function. It is a great helper for many of the machine learning algorithms.

  • I train and validate my model with the normalized data (MinMaxScaler) and save my model.
  • A new input data comes in and I want to use my saved model to make predictions.
  • I no longer have access to the training/test data at this point. All I have is the new single row of input data.
  • How will I normalize this single vector of input data so that it can be fed to my model?

The only normalization that I can think of simply linear transformation of data (e.g. simply map values from [40, 70] range to [-1, 1]). I need a normalization technique that doesn't depend on the full range of data.

Thanks!

$\endgroup$
2
$\begingroup$

You need to normalise the input in the same way that the training data was normalised -- however, you don't need access to this training data during predictions of new data. If you have used a MinMaxScaler for example, then you can re-use this to transform your new data point:

scaler = MinMaxScaler()
X_normalised = scaler.fit_transform(X)

# do other stuff here like train & validate your model

# now we have a new test data point from somewhere, we scale
# it using the scaler we fitted on the training data
new_test_point_normalised = scaler.transform(new_test_point)

# now we can classify it with our model!

You can serialise the scaler to disk using pickle and re-load it when predicting new data points if necessary (i.e., the model training and testing happen in two different scripts).

$\endgroup$
  • 1
    $\begingroup$ That makes so much sense, it never crossed my mind to pickle the scaler as well! It adds a dependency to the model (scaler) but I guess this is the right way to go! Thanks! $\endgroup$ – Guven May 5 '18 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.