I read many time that for machine learning and data mining algorithms the multi-dimensional input data should be scaled (e.g. normalized or standardized). Now my question is whether the average, min or max value shall be calculated for each row (variable) or for the dataset as a whole.

Let's say we have 3 variables

X=[x_1, x_2,.., x_n]

Y=[y_1, y_2,...,y_n]

Z= [z_1, z_2,...,z_n]

I use the following code in python

def standardize (data, train_split):
    data_mean = data[:train_split].mean(axis=0)
    data_std = data[:train_split].std(axis=0)
    return (data - data_mean)/ data_std

def normalize(data, train_split):
    data_max= data[:train_split].max(axis=0)
    data_min = data[:train_split].min(axis=0)
    return (data - data_min) / (data_max -  data_min)

If I saw it correctly, the code (that I copied from a tutorial) calculated the min,max, average and standarddeviation for the whole dataset. Meaning that the maxium value is the maxium of X,Y and Z. Is this correct to do or shall one only normalize and standardize a variable by considering its own values? So the min, max etc. should be calculated for every variable.

I'd appreciate every comment.


Both can make sense depending on your data :

  • Example 1 : if you have a feature that varies between -100 and -50 on the whole dataset you would want to normalize that column based on the values on the whole dataset
  • Example 2 : if each sample is a list of values from different sensors and you only care about their relative values (for anomaly detection for instance) then you might want to normalize each sample

That being said, I think 9 times out of 10 you will want to normalize based on the statistics of the whole dataframe.

Note also that normalizing each sample based on the statistics of the whole dataset is an operation that can be reversed using the normalized data and the dataset statistics and therefore no information is lost : your data is basically the same but rescaled. A neural network could very well learn to reverse this operation.

Normalizing each sample based on this sample statistics can also be useful but this operation actually transforms the representation of your data. As an example, let's say you normalize each sample to a norm of 1, you can't recover the initial data without knowing the initial norm of that specific sample.

  • $\begingroup$ Thanks mprouveur for your answer (I upvoted it). I do not understand your first example. So when the values are between -100 and - 50 I should normalize based on the whole dataset? Why -100 and - 50? It is quite unlikely that the values are within these ranges? Or are you generally referring to variables with only negative values? $\endgroup$ – PeterBe Oct 20 '20 at 16:40
  • $\begingroup$ Well that depends on what type of algorithms you use. If you use regression/decision trees, scaling these values won't affect the model. However if you use a neural network, i will perform best if each feature given as input are within the same range, therefore it is quite common to scale them in the range [-1, +1]. In that sense if you have a feature that varies between -100 and -50 whereas your other features are within the "standard" range of [-1 +1], you will want to scale it down to [-1 ,+1]. $\endgroup$ – mprouveur Oct 21 '20 at 6:41
  • $\begingroup$ Thanks mprouveur for your answer. Unfortunately I do not understand your last comment. Would you mind elaborating on that. Basically there are 2 questions: 1) When shall I standardize and when normalize? 2) When shall I use the whole dataset for normalizing or standardizing and when shall I only use the columns for doing that? I'd appreciate every further comment from you. $\endgroup$ – PeterBe Oct 21 '20 at 7:37
  • $\begingroup$ 1) When shall I standardize and when normalize? A good resource on the topic : medium.com/@dataakkadian/… "Normalizing the data is sensitive to outliers, so if there are outliers in the data set it is a bad practice. Standardization creates a new data not bounded (unlike normalization)." $\endgroup$ – mprouveur Oct 21 '20 at 9:39
  • $\begingroup$ 2) When shall I use the whole dataset for normalizing or standardizing and when shall I only use the columns for doing that? By default I would standardize each column (instead of normalization because of outliers cf the link above), most algorithms perform better when each column of your dataset has a distribution similar to a a normal distribution. That's the default approach, however let's say that each row is the record of a sensor and each column is the value of that sensor at a specific timestamp : then it might (or not) make sense to standardize on each row instead of each column $\endgroup$ – mprouveur Oct 21 '20 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.