I'm trying to predict simple one feature time series data with shifted train data. The source looks like this:

   DATE              PRICE
0  1987-05-20        18.63
1  1987-05-21        18.45
2  1987-05-22        18.55
3  1987-05-25        18.60
4  1987-05-26        18.63

Actual code with data link download gisted here: gist

So the main problem is that it actually can't predict next steps. Roughly speaking: y_train "shifted" relatively to X_train by timesteps defined in parameters. So we getting for X_train and y_train something like this:

timesteps = 5
data = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
# After manipulations which you can find in gist we getting this:
X_train =
    [[ 1  2  3  4  5]
     [ 2  3  4  5  6]
     [ 3  4  5  6  7]
     [ 4  5  6  7  8]
     [ 5  6  7  8  9]
     [ 6  7  8  9 10]
     [ 7  8  9 10 11]
     [ 8  9 10 11 12]
     [ 9 10 11 12 13]
     [10 11 12 13 14]]
y_train =
    [[ 6  7  8  9 10]
     [ 7  8  9 10 11]
     [ 8  9 10 11 12]
     [ 9 10 11 12 13]
     [10 11 12 13 14]
     [11 12 13 14 15]
     [12 13 14 15 16]
     [13 14 15 16 17]
     [14 15 16 17 18]
     [15 16 17 18 19]]

So it is fair to assume that after training LSTM model with X_train (as input) and y_train (as output) we getting model which able to forecast n timesteps ahead. BUT I encountered a problem that trained model not predicting anything - only "duplicates" X_test data. For the convenience I rebuild X_test data and plot it with y_test data which returns from model.predict(): plot of X_test and y_test

So this is result which also contains 'Dataset prices' (pure data from dataset[upper_train + timesteps:]) for clarity.

I can not find where I made a mistake (or maybe this approach is bad?) so I will be grateful for any help!

Unfortunately it is more likely that this approach itself is bad. It's not the fault of your LSTM or neural netowrk.

You may be able to find a lot of online tutorials using RNN/LSTM to predict stock price/crude oil price/bitcoin price or whatever price, based on its price history, and their results are almost surely meaningless.

Note that I am not saying analyzing a market price is meaningless, but predicting future prices based only on historical prices only is usually meaningless, given that the market is well established and most relevant information influencing its price is readily available; or in other words, the market is efficient.

Outside of the domain of data science, the assumptions for the efficient-market hypothesis are not entirely true for most markets, but its consequence is close to the truth for large markets: the asset price follows a random walk. In terms of prediction into the future, the price behaves as a martingale, i.e. the best prediction for tomorrow's price is today's price.

  • Thanks fo reply. Yes I know that this is almost useless. I just practice and trying to dive into ML. But my question is more technical because I can't understand why it happens ( why training with shifted data not giving expected result when predicting, why output data not shifted as well as y_train) – GeorgeMA Sep 20 at 18:53

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.