I am trying to made a model for neural network that tries to predict prices of the stocks point by point using a LSTM (yeah I know that probably did not get anything and I should predict up/down or something realted boolean decision but I like to try this first)
I wil use as features HIGH, LOW, OPEN of last 60 days and try to predict ADJ_CLOSE of next day.
The probem come when I need to normalize the data. I am using data from AAPL from 1980 to 2019 the main problem is that in 80's stocks prices are in pennys and in 10's they are in houndreds of dollars, so I decided to do window normalization using MinMaxScaler from Sklearn.
There is no problem with the train data, I have 7887 data points so I scaled them by windows of 500 and last remaing separetly. The code is like this
x_train = train.values window_size = 500 for i in range(0,7500,window_size): x_train[i:i+window_size] = min_max_scaler.fit_transform(x_train[i:i+window_size]) x_train[i+window_size:] = min_max_scaler.fit_transform(x_train[i+window_size:])
The problems comes when testing I do not know that doing something like
x_test = test.values x_test = min_max_scaler.fit_transform(x_test)
is kind of chating because they are fiting by themselves. But I can not do
x_test = test.values x_test = min_max_scaler.transform(x_test)
after window normalization of train data(using last fitted scaler) because the range between min and max on last window of train data difers a lot from the range in test data and in that case the normalization would not be well done and the test data would not fit between [0,1].