1
$\begingroup$

I have a dataset with different features where some of them are not categorical, so they need to be scaled or normalized (especially the target).

However, normalizing between 0-1 for instance means that the variable maximum value will be equal to one, and the mean to 0.

Now if I receive a new example never seen before, and this example has a value higher than the max of the training examples, how should this value be normalized ?

EDIT

As an example. If my maximum value is 150, it will be scaled to 1.0. Now if I receive a new example, with a value equal to 320, how should it be scaled ?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Possible duplicate of Should we apply normalization to test data as well? $\endgroup$
    – yoav_aaa
    Nov 25, 2019 at 14:42
  • $\begingroup$ What do you exactly mean by 'not knowing the maximum'? If it's a case of missing values, you should handle imputation first or discard those records. $\endgroup$
    – Piotr Rarus
    Nov 26, 2019 at 8:46

1 Answer 1

Reset to default
2
$\begingroup$

If your model works in production, you should not retransform your scaler, you should transform new example like 150 is still the maximum value. (It will give you higher than 1, so its a bit problematic but possible solution is below) However you can still label those example as outlier.

Possible solution for that case: If you have high number of outliers/leverages, you should consider tree ensembles and/or regularized models.

If your predictor is not on production, just add those examples to your train set and fit again since your sample in the first training would be different than the reality.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.