Scaling of variables considering the values of a single column or the whole dataset

Question

I read many time that for machine learning and data mining algorithms the multi-dimensional input data should be scaled (e.g. normalized or standardized). Now my question is whether the average, min or max value shall be calculated for each row (variable) or for the dataset as a whole.

Let's say we have 3 variables

X=[x_1, x_2,.., x_n]

Y=[y_1, y_2,...,y_n]

Z= [z_1, z_2,...,z_n]

I use the following code in python

def standardize (data, train_split):
    data_mean = data[:train_split].mean(axis=0)
    data_std = data[:train_split].std(axis=0)
    return (data - data_mean)/ data_std


def normalize(data, train_split):
    data_max= data[:train_split].max(axis=0)
    data_min = data[:train_split].min(axis=0)
    return (data - data_min) / (data_max -  data_min)

If I saw it correctly, the code (that I copied from a tutorial) calculated the min,max, average and standarddeviation for the whole dataset. Meaning that the maxium value is the maxium of X,Y and Z. Is this correct to do or shall one only normalize and standardize a variable by considering its own values? So the min, max etc. should be calculated for every variable.

I'd appreciate every comment.

Answer 1

Both can make sense depending on your data :

• Example 1 : if you have a feature that varies between -100 and -50 on the whole dataset you would want to normalize that column based on the values on the whole dataset
• Example 2 : if each sample is a list of values from different sensors and you only care about their relative values (for anomaly detection for instance) then you might want to normalize each sample

That being said, I think 9 times out of 10 you will want to normalize based on the statistics of the whole dataframe.

Note also that normalizing each sample based on the statistics of the whole dataset is an operation that can be reversed using the normalized data and the dataset statistics and therefore no information is lost : your data is basically the same but rescaled. A neural network could very well learn to reverse this operation.

Normalizing each sample based on this sample statistics can also be useful but this operation actually transforms the representation of your data. As an example, let's say you normalize each sample to a norm of 1, you can't recover the initial data without knowing the initial norm of that specific sample.