# What is the best way to normalize a set of datasets

I have a data set that contains the same Time series "Sensor readings" for different days and I want to make a deep learning model to predict these values. What I did was I splatted the data into Time series data according to the day, then I normalized it separately (min-max) (the readings have different ranges, for example, the max value for the first day is 100 but the max for second is 48) but I'm really confused now do I need to normalize it using the max/min of the all days or what I did was right?

• if you want to train one model for all time series then better to normalise all at once, not one-by-one Jul 16, 2020 at 12:03
• what I thought is dealing with every Time series as separate one
– Izo
Jul 16, 2020 at 12:15

If you know strict bounds on the sensor output, that would be better than normalizing by the min/max of the dataset. Even if the bounds are not necessarily strict, but simply reasonable, that would suffice. For example, if there are no theoretical bounds on a temperature sensor, you might reasonably impose strict bounds given prior knowledge about its environment (e.g. if a temp sensor was placed in NY, you might assume strict bounds as -30C to 50C)

If you were to normalize by the min/max of the training data, what do you expect to happen if the deployed model encounters a value outside this range? If, for example, your training set had min 5 and max 30, how would you normalize an input of -10? It would be much more intuitive and reliable to shift up to a nonnegative domain, (i.e. subtract your strict minima), then scale to a value in [0,1] via the strict range.

Also, depending on your neuronal activation functions, consider centering your data in addition to scaling it.

You should apply and normalize using the total min/max including all the historical data in your dataset. Your model expects the same normalization within each feature across all measurements in that feature. For example

sensor_1_day_1 -> 0, 1, 2, 2, 3 sensor_1_day_2 -> 0, .1, .3, .4, .1

normalize sensor_1 for both days with [min,max] of [0,3] and normalize

sensor_1_day_1_norm -> 0/3, 1/3, 2/3, 2/3, 3/3 sensor_1_day_2_norm -> 0/3, .1/3, .3/3, .4/3, .1/3

Don't forget to de-normalize the predictions (multiply by 3 in this example) as they will also be normalized. Side note: If you apply a different normalization to each day, and have to de-normalize each day differently, this would be very complex to remember and handle.

• you're right it's really complex when different normalization been applied
– Izo
Jul 16, 2020 at 15:12