I am trying to made a model for neural network that tries to predict prices of the stocks point by point using a LSTM (yeah I know that probably did not get anything and I should predict up/down or something realted boolean decision but I like to try this first)

I wil use as features HIGH, LOW, OPEN of last 60 days and try to predict ADJ_CLOSE of next day.

The probem come when I need to normalize the data. I am using data from AAPL from 1980 to 2019 the main problem is that in 80's stocks prices are in pennys and in 10's they are in houndreds of dollars, so I decided to do window normalization using MinMaxScaler from Sklearn.
There is no problem with the train data, I have 7887 data points so I scaled them by windows of 500 and last remaing separetly. The code is like this

x_train = train.values 

window_size = 500
for i in range(0,7500,window_size):
    x_train[i:i+window_size] = min_max_scaler.fit_transform(x_train[i:i+window_size])

x_train[i+window_size:] = min_max_scaler.fit_transform(x_train[i+window_size:])

The problems comes when testing I do not know that doing something like

x_test = test.values
x_test = min_max_scaler.fit_transform(x_test)

is kind of chating because they are fiting by themselves. But I can not do

x_test = test.values
x_test = min_max_scaler.transform(x_test)

after window normalization of train data(using last fitted scaler) because the range between min and max on last window of train data difers a lot from the range in test data and in that case the normalization would not be well done and the test data would not fit between [0,1].

Any help?


1 Answer 1


You should always fit the scaler with the train data. This is one of the typical validation problems, it is hard to replicate train test conditions.

Have you considered doing rolling/rollout window split? Roll out window

This way the range of values will be closer between train and test.

THe problem of this is that you loose some of the training information. On the other hand information about 40 years ago of stock has not much to do when forecastin the present

  • $\begingroup$ But with this methodology probably I would get better results but my I did not solve mi problem. My main question is should I scale test data witth a scaler fitted with train data or should I use test data to fitted it? $\endgroup$
    – lujoselu
    Commented Apr 15, 2020 at 22:57
  • 1
    $\begingroup$ @lujo You should always use a scaler fitted in the train data. With this methodology you will be able to have less variance between train and test $\endgroup$ Commented Apr 16, 2020 at 6:52
  • $\begingroup$ Yes but then I have got the same problem whats happen I have diferent range in train data ans in test data. In that case if I transform test data with a min scaler fitted on train data, test data would not scale between 0 and 1, and model would not work fine over them. $\endgroup$
    – lujoselu
    Commented Apr 16, 2020 at 18:00
  • $\begingroup$ Consider scaling as another machine learning algorithm. You can´t train with the test data no matter what $\endgroup$ Commented Apr 17, 2020 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.