Let be $X = (X_1,...., X_n)$ an univariate time serie. I would like to know how to standardize my data when I split it into train and test data. Let me explain you how I tranform $X$ so that I can fit an LSTM neural net. From $X$ I make a new input data and its corresponding output data. So, we have: $X = ( (X_1, ..., X_m), ... , (X_{n-m}, ..., X_{n-1}) )$
$Y = (X_{m+1}, ..., X_n)$
$\text{Card}X = \text{Card}Y$
Let's set $p$ the size of my test set. If I use Python's notation, we have:
$X_{train} = X[:-p]$
$X_{test} = X[-p:]$
Idem for $Y$ ... Now, I am wondering how to standardize my data. I think that standardizing $X$ before splitting the data into train and test sets could lead to over-fitting since we a transformation that involves all $X_i$. Basically, I am not sure that the sum (mean, standard deviation) will drown the information. In this case I think it could be better to just compute the mean and the standard deviation in the training set and use them to standardize both of the train and test sets. For me it makes no sense to standardize them separately since $\text{Card}X_{test} << \text{Card}X_{train}$. But may be I am wrong. I would also like to know whether I have to standardize $Y$ and $X$ or just $X$. When I am working with MLP neural net I used to just normalize the input data..

So, thank you first for reading and if you have any ideas or remarks, any questions to ask, please let me know. I can explain more it is up to you :)

P.S. I don't find a 'standardization' tag and I thus use the tag named normalization.


Welcome kakarotto! The first thing, standarize with respect to the training set only, then use the statistics to standarize the other sets. As a rule of thumb, we should keep in mind that the test set does not exits while training... (although we have access to it) When training, if the network does not see the test set we (the humans) do either…

If the test set was well selected, and so the whole dataset, the training set and the test set should come from the same probability density function and, if enough data is provided, the statistics you get from the training set would be close to the statistics of the pdf (assuming unbiased estimators for these statistics, that should be the case for the mean and standard deviation)

Based on my experience, only standarize the input data. With the output data, I usually scale it (min-max scaler) if I know the boundaries (it they even exist…). Another usefull trick that usually worked for me its to not feed the net with the input data rather with its relative differences, for example:

$X = (0, X_1 - X_0, X_2 - X_0,\dots,X_m-X_0)$

| improve this answer | |
  • $\begingroup$ Thanks :)That is what I think but I am not sure that my dataset verifies those hypothesis. I mean, let's imagine that I have 5-year sale records and I want to predict the sales monthly. Let's say 3 months in the future. So, in this case, I may have a training set that includes the first 33 months and test having the 3 last months. You can guess that the nature of my distribution is completely random. In this case, how should I standardize ? Is it a good way to solve a time serie forecasting problem ? I am not familiar with time series, so I may be totally wrong ... $\endgroup$ – kakarotto Jul 8 '19 at 11:26
  • $\begingroup$ When working with time series you have to indentify if your data is stationary (i guess no) and handle the trend and the seasonality, if it is posible and they eve exist... When taking about prices is much far logical to work with relative prices and relative differences (for example the difference with respect to the beginning of the year/month…) since prices increase with time... I'll give you a good link machinelearningmastery.com/… $\endgroup$ – ignatius Jul 8 '19 at 11:32
  • $\begingroup$ adding to @ignatius answer, using those differences implies using the derivative of the timeseries instead of the original signal. Differentiation is a very dangerous operation in the context of timeseries, because it amplifies noise and can ruin your model. In some cases it might work, but it is safe to say that in general it is considered a black sheep, especially when measurements are sensor data. In fact, integration of timeseries might be better than differentiation, because it smoothens the signal like a low-pass filter. $\endgroup$ – pcko1 Jul 8 '19 at 13:45
  • $\begingroup$ I know that, e.g., machinelearningmastery.com recommends differentiating timeseries to remove trends and make the signal stationary but take this with a pinch of salt. $\endgroup$ – pcko1 Jul 8 '19 at 13:51
  • $\begingroup$ @pcko1 that's really interesting. If I have a correct understanding, integrating a my time serie means to apply such a transformation : F(t0) = sum{t <= t0) F(t). So, assuming an additive model, we have : F(t0) = sum{t<+t0} Trend(t) + ... If the trend increase over the time, the integration will lead to increase the trend. I see a problem, don't you ? Thanks. $\endgroup$ – kakarotto Jul 8 '19 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.