While designing a ML model, how do I decide if I need to go for Normalization and not Standardization or vice-versa? On what factor is this decision made?


Before we start keep in mind that in most cases it doesn't play much of a difference which of the two you'll choose.

Now to answer your question, generally speaking the choice should be made based on what model you want to employ:

  • If you use a distance-based estimator (e.g. k-NN, k-means) it's better to normalize your features so that they occupy the same exact range of values (i.e. $[0,1]$). This forces your estimator to treat each feature with equal importance.

  • If you're using Neural Networks, it's better to standardize your features, because gradient descent has some useful properties when your data is centered around $0$ with unit variance.

  • Tree-based algorithms don't require any form of scaling, so its irrelevant if you scale or normalize your features.

As a rule of the thumb, I usually standardize the data (unless I'm going to strictly work with distance-based algorithms).


I think it purely depends upon the model. For instance, if it is a Naive Bayes, as it deals with probabilities only, you can't use the negative values. In this case Normalization works!

When you deal with geometry based algorithms such as SVM or Logistic Regression, it's better to standardize the data because due to (-1,1) symmetry in the data. The learning of training process happens very fast (due to symmetry points) when compared to Normalization.

I believe Standardization mostly works for many algorithms. However, what I suggest you is do check the context of algorithm and loss function metric.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.