# How to normalize a data set of multiple time series?

I have the a data set representing the electricity consumption of 25 000 customer. The electricity readings are taken from each smart meter each 15 min for a period of 3 days. The data is takes from counters of residential, commercial and Industrial buildings thing that makes us have different consumption modes. Each record in the dataset is in the format:

meter_id timestamp cumulative_value


using pandas function groupby('meter_id') and by transforming the each group to an array format I get the array format that corresponds to an independent time series ready to feed to a keras LSTM model.

The problem that I have is that my model does not converge no matter the learning rate is. I tried all kind of normalization and standardization but still having this over-fitting problem. Displaying the probability distribution of all the consumption values is as follows:

Here is also the histogram after normalization:

My LSTM network configuration is as follows:

model_cum = Sequential()


and the output I get is the following:

Train on 15281 samples, validate on 1698 samples
Epoch 1/20
15281/15281 [==============================] - 118s 8ms/sample - loss: 0.4545 - mean_squared_error: 0.2075 - val_loss: 0.4467 - val_mean_squared_error: 0.1998
Epoch 2/20
15281/15281 [==============================] - 117s 8ms/sample - loss: 0.4378 - mean_squared_error: 0.1926 - val_loss: 0.4301 - val_mean_squared_error: 0.1852
Epoch 3/20
15281/15281 [==============================] - 117s 8ms/sample - loss: 0.4213 - mean_squared_error: 0.1784 - val_loss: 0.4134 - val_mean_squared_error: 0.1711
Epoch 4/20
15281/15281 [==============================] - 116s 8ms/sample - loss: 0.4045 - mean_squared_error: 0.1646 - val_loss: 0.3966 - val_mean_squared_error: 0.1575
Epoch 5/20
15281/15281 [==============================] - 117s 8ms/sample - loss: 0.3876 - mean_squared_error: 0.1512 - val_loss: 0.3797 - val_mean_squared_error: 0.1444
Epoch 6/20
15281/15281 [==============================] - 115s 8ms/sample - loss: 0.3706 - mean_squared_error: 0.1384 - val_loss: 0.3626 - val_mean_squared_error: 0.1317
Epoch 7/20
15281/15281 [==============================] - 116s 8ms/sample - loss: 0.3535 - mean_squared_error: 0.1260 - val_loss: 0.3453 - val_mean_squared_error: 0.1194
Epoch 8/20
15281/15281 [==============================] - 116s 8ms/sample - loss: 0.3362 - mean_squared_error: 0.1141 - val_loss: 0.3277 - val_mean_squared_error: 0.1076
Epoch 9/20
15281/15281 [==============================] - 115s 8ms/sample - loss: 0.3183 - mean_squared_error: 0.1024 - val_loss: 0.3098 - val_mean_squared_error: 0.0962
Epoch 10/20
15281/15281 [==============================] - 115s 8ms/sample - loss: 0.3002 - mean_squared_error: 0.0912 - val_loss: 0.2916 - val_mean_squared_error: 0.0852
Epoch 11/20
15281/15281 [==============================] - 116s 8ms/sample - loss: 0.2818 - mean_squared_error: 0.0805 - val_loss: 0.2730 - val_mean_squared_error: 0.0747
Epoch 12/20
15281/15281 [==============================] - 115s 8ms/sample - loss: 0.2635 - mean_squared_error: 0.0706 - val_loss: 0.2540 - val_mean_squared_error: 0.0647
Epoch 13/20
15281/15281 [==============================] - 116s 8ms/sample - loss: 0.2441 - mean_squared_error: 0.0608 - val_loss: 0.2346 - val_mean_squared_error: 0.0553
Epoch 14/20
15281/15281 [==============================] - 114s 7ms/sample - loss: 0.2243 - mean_squared_error: 0.0516 - val_loss: 0.2147 - val_mean_squared_error: 0.0463
Epoch 15/20
15281/15281 [==============================] - 115s 8ms/sample - loss: 0.2037 - mean_squared_error: 0.0428 - val_loss: 0.1943 - val_mean_squared_error: 0.0380
Epoch 16/20
15281/15281 [==============================] - 115s 8ms/sample - loss: 0.1832 - mean_squared_error: 0.0349 - val_loss: 0.1731 - val_mean_squared_error: 0.0302
Epoch 17/20
15281/15281 [==============================] - 115s 8ms/sample - loss: 0.1624 - mean_squared_error: 0.0278 - val_loss: 0.1513 - val_mean_squared_error: 0.0231
Epoch 18/20
15281/15281 [==============================] - 116s 8ms/sample - loss: 0.1396 - mean_squared_error: 0.0210 - val_loss: 0.1287 - val_mean_squared_error: 0.0168
Epoch 19/20
15281/15281 [==============================] - 115s 8ms/sample - loss: 0.1168 - mean_squared_error: 0.0152 - val_loss: 0.1051 - val_mean_squared_error: 0.0113
Epoch 20/20
15281/15281 [==============================] - 114s 7ms/sample - loss: 0.0932 - mean_squared_error: 0.0103 - val_loss: 0.0808 - val_mean_squared_error: 0.0067


There mean_square error loss in this case is always under zero and nearly fixed. What is the problem with what I am doing and how to solve it ?

You could try using the box cox transformation on your target variable to make it normal-distributed, and repeat the same process.

Then you will have to transform the values of the predictions with the inverse transformation . The two functions are on scipy.

from scipy.special import  inv_boxcox
from scipy.stats import boxcox

• In case this works, how to denormalize back the data to make predictions? – pentanol Sep 24 '19 at 13:00
• @pentanol with the inverse transformation ? – Duccio Piovani Sep 24 '19 at 13:03
• Yes, I mean what is the inverse of box cox transfrom – pentanol Sep 24 '19 at 13:07
• docs.scipy.org/doc/scipy/reference/generated/… I will edit the answer – Duccio Piovani Sep 24 '19 at 13:11
• The idea was good, the the dataset has the shape of of normal distribution but I still suffer from underfitting as mentioned above. – pentanol Sep 24 '19 at 13:59

The solution is simple. The model is underfitting, that means it cant learn from the data. The solution is to increase the number of LSTM layers and increase the Dropout to avoid overfitting. For more details you can see the following link