# Why are predictions from my LSTM Neural Network lagging behind true values?

I am running an LSTM neural network in R using the keras package, in an attempt to do time series prediction of Bitcoin. The issue I'm running into is that while my predicted values seem to be reasonable, for some reason, they are "lagging" or "behind" the true values. Right below is some of my code, and farther down I have some graphs to show you what I mean. My model code:

batch_size = 2

model <- keras_model_sequential()

model%>%
layer_lstm(units=22,
batch_input_shape = c(batch_size, 1, 22), use_bias = TRUE, stateful = TRUE,
return_sequences = TRUE) %>%
layer_lstm(units=16, batch_input_shape = c(batch_size, 1, 22), stateful = TRUE, return_sequences = TRUE) %>%
layer_dense(units=1)
model %>% compile(
loss = 'mean_absolute_error',
optimizer = optimizer_adam(lr= 0.00004, decay = 0.000004),
metrics = c('mean_absolute_error')
)
summary(model)

Epochs <- 50
for (i in 1:Epochs){
print(i)
model %>% fit(x_train, y_train, epochs=1, batch_size=batch_size, verbose=1, shuffle=FALSE)
model %>% reset_states()
}


So in case that's not clear, I have a neural network with 1 middle layer - I have 22 units in the input layer (equal to my number of variables), 16 units in middle layer and my one output layer.

Here is a graph of the training data fit (blue is fit, red are true values):

I am predicting the Bitcoin price 24 hours ahead. I have hourly data, so I'm doing this prediction by simply shifting the Bitcoin price column of my data 24 steps back, so I'm matching past predictor conditions with future outcome.

From the picture above, you can see that the training fit is very strong. However, take a look at my out-of-sample predictions vs. true values (again, blue line is model prediction, red line is true value):

At first glance, it's really not shabby. However, if you take a closer look (and it becomes VERY obvious when I zoom in to smaller time scales), the predicted blue line often lags behind the "true" red line:

The odd thing to me is that this is not a consistent problem. If you look at some of the movements towards the right of the graph, the model gets it on-target (no lag). Additionally, from zooming in and really looking carefully, I have found that the apparent lag itself is not consistent in magnitude, ranging from around 14 hours to sometimes 22 hours (meaning the prediction is barely usable, since it's predicting 24 hours ahead, but "lagging" the true value by 22 hours, so I'm really only getting 2 hours ahead of real prediction).

I have tried increasing my batch size (to 5, 10, 30), which doesn't make the problem any better (might even make it worse). I tried increasing the size of my middle neuron layer (to 20, 30, 44), which also didn't fix the issue. Having the loss function as "Mean Absolute Error" SEEMS to work better than "Mean Squared Error", but what you're looking at is already the MAE version, so the problem obviously persists.

Around half of my inputs into the neural network model are lagged values of the Bitcoin price (BTC price 24 hours ago, 25 hours ago etc.), so I thought maybe the problem was that my model was simply grabbing those past values and replicating them because the model couldn't find any other meaningful connections to my predictors. However,

1. You can see that the problem doesn't exist in the training dataset fit, so I don't think this is an issue of my model only using past price values as its best guess.
2. I tried changing the past lags that were used (for example, instead of using the 24 hour ago value, I used the 30 hour ago value). However, this didn't make a difference, so I'm pretty confident now that the issue is not that my model is relying solely on past price values..

As a result, I really have no idea where this gap is coming from.

Any advice, suggestions or tips would be appreciated on how I could deal with this odd gap. Thank you very much!

EDIT (please read entirely, important): In order to once-and-for-all test the idea that it's the lagged time series inputs causing the issue, I just ran the neural network with all past values of the price removed. As in, ALL of the inputs were exogenous variables, no time-series lagged values, and while it's a bit hard to tell (because the predictions are messier), the problem appears to persist. Take a look:

I think this pretty much definitively proves that the lag is not coming from past price values being replicated. HOWEVER, I looked at the training data fit for the model with no time-series inputs, and it is obvious that it ALSO has an offset/lag. Example:

One more thing I need to mention. When I run this neural network on the same data but without offsetting the predictors from outcomes, there is no problem. That's to say, when I run the data without shifting the Bitcoin price column back, meaning my network is matching current conditions to current price, this prediction offset does not exist. In fact, I've been playing around with this offset (so, trying to predict 12 hours ahead, 24 hours ahead, 48 and 72 hours ahead) and it seems like changing this changes the lag in prediction. I have no idea why. When I change it to predicting 72 hours ahead, the prediction lag isn't exactly 72 hours (just like it isn't exactly 24 hours when I'm predicting that far ahead). However, the prediction lag noticeably increases/decreases when I increase/decrease how far ahead I'm trying to predict..

EDIT 2: I am now quite certain that I'm making some mistake in my data processing. Since I noticed that the prediction offset increases/decreases with how far ahead I'm trying to predict, I tried making the value for "how many hours ahead I want to predict" negative. (-20, to be exact). And here's what I saw now:

Sure enough, the predictions are now significantly "ahead" of the actual values. As a result, I think I'm making some kind of basic data processing error. As of right now, I haven't found the error yet though.

Welcome to the site.

I think you were right that the prediction lags behind the true value because the series is autoregressive (i.e. a strong way to predict tomorrow’s value is “It will be about the same as today”). Your model therefore corrects itself with the new information when it misses a big jump. In other words, if the price jumps one day and your model does not predict that, it has learned to take into account the higher price for the prediction of the next day’s price.

In response to your numbered points above:

1. Is this based on eyeballing the data? Can you show us any results that prove the model behaves differently during training?
2. Are you sure that when you change the lag to 30 hours, the lag in prediction doesn’t just change to a 30-hour lag, as we’d expect from an autoregressive model?

I recommend using another model as a baseline (e.g. Facebook Prophet) and seeing whether the values your model is producing that are significantly different to that of the baseline model are more correct. This gives you a more rigorous alternative to troubleshooting your data by eye. Where your model is less accurate, you can look at the type of inputs at those time steps.

• 1. Well, wouldn't you say that the gap that is apparent in testing is absent in the training fit graph? (first picture in my answer). Or is this not sufficient proof? 2. I'll take another look, but I'm pretty confident (additionally, notice that the lag is not consistent, nor is it equal to 24 hours even right now). – Vladimir Belik Jun 29 '20 at 13:10
• @VladimirBelik It's hard to tell on the training data because the time scale means the gap would only be about 1 pixel or less. But if you look closely, there are several obvious 1-pixel gaps. – user253751 Jun 29 '20 at 13:11
• @user253751 I just zoomed in to the training fit, and I see that there is not a consistent gap. The gap is there every once in a while, but again, it's not a consistent size and isn't always there. – Vladimir Belik Jun 29 '20 at 13:15
• Additionally, I just want to emphasize that the "most recent" lagged value the model has as input is the price 25 hours ago. However, the lag sometimes is 14 or 17 hours - this means that the lag occurs even when the model hasn't "seen" the past values that would allow it to "correct" itself by replicating those past values. As a result, I'm really having a hard time seeing how this can come from those past time-series values. – Vladimir Belik Jun 29 '20 at 13:28
• @user253751 Please take a look at the edit I made at the end of my question. What do you think? I might be overfitting (though what would be the evidence of that?), but it's pretty clear that it's not the time series values causing the issue. – Vladimir Belik Jun 29 '20 at 16:32

Welcome to Data Science on Stack Exchange,
This is a common question, predicting future prices or forecasting. The gap you see is due to the random nature of prices such as this, along with the underlying complexity of this topic. Unless there is a time pattern in the data, a LSTM model won't predict well. LSTM will especially perform poorly if the data is changing direction often, going up and down in value.
A lot of discussion goes around which model you should use, but not sure any one of them are consistently the best. For some general ideas on different techniques, applied to the stock market in this case, here is a good reference.

https://www.analyticsvidhya.com/blog/2018/10/predicting-stock-price-machine-learningnd-deep-learning-techniques-python/

It mentions LSTM (popular if you believe in momentum), and also Arima, FBProphet, etc.

However, there are usually many other variables that have a large influence on the future price that you won't get this way, such as sentiment, news articles or announcements, etc. You might look at using an ensemble, maybe LSTM + 1 or 2 other models, and combine these different inputs so they can be included in your prediction.

• Well, for my LSTM model, I have around 10 time series inputs (lagged price values), and then the other 12 variables are various exogenous variables that I believe to be related to the price. I'll definitely read the article, but what exactly do you suggest? I have not had any success with any other type of model (ARIMA, ARCH/GARCH), so I honestly don't know if there is strong underlying time series structure to this data. (Also - isn't FBProphet just the same as doing an ARIMA model?) – Vladimir Belik Jun 29 '20 at 13:21

I've been fiddling a bit with LSTMs myself to predict windspeeds using inertial drone data and some of my plots had a similar "offset" to yours. Have you scaled your inputs using a MinMax or Standard scalar? I've also have a surprisingly good amount of success implementing a KNN algorithm to predict the windspeeds with mean bias errors oftentimes lower than those found by LSTM.

• Interesting! I scaled my inputs by normalizing them to a mean of 0 and standard deviation of 1 (multiplied each input variable value by that column's standard deviation and subtracted the column's mean). Is this a bad/weird approach? Would it make a difference? Also, isn't KNN for classification? I have never used it or learned about it, so forgive me if I have no idea what I'm talking about there. – Vladimir Belik Jun 29 '20 at 16:36
• Your scaling seems pretty close to the standard scaling method (basically subtracting the column mean from each input and dividing by the std dev of the column). Depending on the range of your input variables your scaling approach or the standard may not make much of a difference, but data pre-processing is almost always an essential step. KNN is indeed a classification algorithm but can be used to find a fit to training samples and extended to predict on new data. The accuracy may drop a bit but classification is nearly real-time if that is an acceptable trade-off. – BbJug Jun 29 '20 at 18:32
• I'll give that a try if I can't fix my neural network. As I mentioned in my edit, the only thing that appears to affect the prediction offset is how far forward I'm trying to predict. As a result, I have a hard time believing that there's something fundamentally wrong or bad about the model, and I feel like there should be some kind of relatively simple fix. – Vladimir Belik Jun 29 '20 at 18:43