LSTM future steps prediction with shifted y_train relatively to X_train

Question

I'm trying to predict simple one feature time series data with shifted train data. The source looks like this:

   DATE              PRICE
0  1987-05-20        18.63
1  1987-05-21        18.45
2  1987-05-22        18.55
3  1987-05-25        18.60
4  1987-05-26        18.63

Actual code with data link download gisted here: gist

So the main problem is that it actually can't predict next steps. Roughly speaking: y_train "shifted" relatively to X_train by timesteps defined in parameters. So we getting for X_train and y_train something like this:

timesteps = 5
data = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
# After manipulations which you can find in gist we getting this:
X_train =
    [[ 1  2  3  4  5]
     [ 2  3  4  5  6]
     [ 3  4  5  6  7]
     [ 4  5  6  7  8]
     [ 5  6  7  8  9]
     [ 6  7  8  9 10]
     [ 7  8  9 10 11]
     [ 8  9 10 11 12]
     [ 9 10 11 12 13]
     [10 11 12 13 14]]
y_train =
    [[ 6  7  8  9 10]
     [ 7  8  9 10 11]
     [ 8  9 10 11 12]
     [ 9 10 11 12 13]
     [10 11 12 13 14]
     [11 12 13 14 15]
     [12 13 14 15 16]
     [13 14 15 16 17]
     [14 15 16 17 18]
     [15 16 17 18 19]]

So it is fair to assume that after training LSTM model with X_train (as input) and y_train (as output) we getting model which able to forecast n timesteps ahead. BUT I encountered a problem that trained model not predicting anything - only "duplicates" X_test data. For the convenience I rebuild X_test data and plot it with y_test data which returns from model.predict():

So this is result which also contains 'Dataset prices' (pure data from dataset[upper_train + timesteps:]) for clarity.

I can not find where I made a mistake (or maybe this approach is bad?) so I will be grateful for any help!

Answer 1

Unfortunately it is more likely that this approach itself is bad. It's not the fault of your LSTM or neural netowrk.

You may be able to find a lot of online tutorials using RNN/LSTM to predict stock price/crude oil price/bitcoin price or whatever price, based on its price history, and their results are almost surely meaningless.

Note that I am not saying analyzing a market price is meaningless, but predicting future prices based only on historical prices only is usually meaningless, given that the market is well established and most relevant information influencing its price is readily available; or in other words, the market is efficient.

Outside of the domain of data science, the assumptions for the efficient-market hypothesis are not entirely true for most markets, but its consequence is close to the truth for large markets: the asset price follows a random walk. In terms of prediction into the future, the price behaves as a martingale, i.e. the best prediction for tomorrow's price is today's price.

Answer 2

Market is perfect, and this is that causes that model cannot predict. There are plenty of kinda LSTM or other more perfect algorithms that do not allow the prices to behave as a predictable index (if there is some predictable value million of algorithms will take profit and price will move to a no predictable value again).

LSTM does train and fits correctly with trained data, but this information is no use with the test data that comes after.

As a demonstration, I have changed your input data with a predictable periodic signal like a sinus, so LSTM can learn correctly to predict the future from the past with the data shifted as you requested.

sinus_input = [np.sin(2 * np.pi * i/20) for i in range(1275)]
index = [i for i in range(1250)]

sinus_data = pandas.DataFrame(list(zip(index, sinus_input)), columns =['data', 'value']) 

You can download the notebook from here:

https://gist.github.com/m-alcu/0be57483552385ce9a2622eb103ac988