I have a data set that contains the same Time series "Sensor readings" for different days and I want to make a deep learning model to predict these values. What I did was I splatted the data into Time series data according to the day, then I normalized it separately (min-max) (the readings have different ranges, for example, the max value for the first day is 100 but the max for second is 48) but I'm really confused now do I need to normalize it using the max/min of the all days or what I did was right?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ if you want to train one model for all time series then better to normalise all at once, not one-by-one $\endgroup$– Nikos M.Jul 16, 2020 at 12:03
-
$\begingroup$ what I thought is dealing with every Time series as separate one $\endgroup$– IzoJul 16, 2020 at 12:15
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
If you know strict bounds on the sensor output, that would be better than normalizing by the min/max of the dataset. Even if the bounds are not necessarily strict, but simply reasonable, that would suffice. For example, if there are no theoretical bounds on a temperature sensor, you might reasonably impose strict bounds given prior knowledge about its environment (e.g. if a temp sensor was placed in NY, you might assume strict bounds as -30C to 50C)
If you were to normalize by the min/max of the training data, what do you expect to happen if the deployed model encounters a value outside this range? If, for example, your training set had min 5 and max 30, how would you normalize an input of -10? It would be much more intuitive and reliable to shift up to a nonnegative domain, (i.e. subtract your strict minima), then scale to a value in [0,1] via the strict range.
Also, depending on your neuronal activation functions, consider centering your data in addition to scaling it.
$\begingroup$
$\endgroup$
1
You should apply and normalize using the total min/max including all the historical data in your dataset. Your model expects the same normalization within each feature across all measurements in that feature. For example
sensor_1_day_1 -> 0, 1, 2, 2, 3 sensor_1_day_2 -> 0, .1, .3, .4, .1
normalize sensor_1 for both days with [min,max] of [0,3] and normalize
sensor_1_day_1_norm -> 0/3, 1/3, 2/3, 2/3, 3/3 sensor_1_day_2_norm -> 0/3, .1/3, .3/3, .4/3, .1/3
Don't forget to de-normalize the predictions (multiply by 3 in this example) as they will also be normalized. Side note: If you apply a different normalization to each day, and have to de-normalize each day differently, this would be very complex to remember and handle.
-
$\begingroup$ you're right it's really complex when different normalization been applied $\endgroup$– IzoJul 16, 2020 at 15:12