Time series prediction using ARIMA vs LSTM

Question

The problem that I am dealing with is predicting time series values. I am looking at one time series at a time and based on for example 15% of the input data, I would like to predict its future values. So far I have come across two models:

• LSTM (long short term memory; a class of recurrent neural networks)
• ARIMA

I have tried both and read some articles on them. Now I am trying to get a better sense on how to compare the two. What I have found so far:

1. LSTM works better if we are dealing with huge amount of data and enough training data is available, while ARIMA is better for smaller datasets (is this correct?)
2. ARIMA requires a series of parameters (p,q,d) which must be calculated based on data, while LSTM does not require setting such parameters. However, there are some hyperparameters we need to tune for LSTM.
3. EDIT: One major difference between the two that I noticed while reading a great article here, is that ARIMA could only perform well on stationary time series (where there is no seasonality, trend and etc.) and you need to take care of that if want to use ARIMA

Other than the above-mentioned properties, I could not find any other points or facts which could help me toward selecting the best model. I would be really grateful if someone could help me finding articles, papers or other stuff (had no luck so far, only some general opinions here and there and nothing based on experiments.)

I have to mention that originally I am dealing with streaming data, however for now I am using NAB datasets which includes 50 datasets with the maximum size of 20k data points.

Answer 1

Statement 1 is correct, statement 2 is correct, but requires elaboration, and statement 3 is incorrect for seasonal ARIMA:

The following might point you in the right direction but hopefully you'll get a few more answers with more depth in the arena of LSTM.

You mention that you have tried both algorithms and that you are simply trying to figure out which one is better, which leads me to think you may be having more trouble with the data science process and cross validation than with the specifics of the models.

Time series in general:

Time series, in general, are difficult to forecast. If they were easy to forecast then all data scientists would be wealthy, having accurately forecast the value of all of the stocks. The reality is that hedge funds, on average, do not outperform the market and that time series forecasting is typically very poor and applies only to very short durations. The main problems are that there is a lot of noise, there are many hidden influences, models are overly simplistic, influencers do not behave as we think they should, the interplay between linearity and nonlinearity is subtle and confusing, ... ad infinitum.

ARIMA

You are incorrect in your assessment that ARIMA requires stationary time series to forecast on. Non-seasonal ARIMA has three input values to help control for smoothing, stationarity, and forecasting ARIMA(p,d,q), where:

• p is the number of autoregressive terms,
• d is the number of nonseasonal differences needed for stationarity, and
• q is the number of lagged forecast errors in the prediction equation.

By contrast seasonal ARIMA has six input values ARIMA(p,d,q,P,D,Q), where:

• P is the number of seasonal autoregressive terms,
• D is the number of seasonal differences, and
• Q is the number of seasonal moving-average terms.

Subject to the qualifying statements above, I suggest playing with seasonal ARIMA to get a feel for the intricacies involved in smoothing, de-seasoning, de-trending, de-noiseing, and forecasting.

LSTM

I don't know enough about LSTM to add much here. I will add that red flags tend to be raised when someone begins at data science exercise with deep learning. I suggest learning as much as you can using ARIMA and then applying some of your ARIMA expertise to help you learn LSTM. Neural networks can be a very powerful tool, but they:

• can take a long time to run,
• often require more data to train than other models, and
• have lots of input parameters to tune.

Cross validation and comparing models:

Time series are fun in that all training data can usually be turned into supervised learning training sets. Once can simply take a time series and roll back time. That is... pick a point in time and pretend that you don't have any additional data, then produce a forecast and see how well you did. You can march through the time series doing this $n$ times in order to get an assessment of the performance of your model and to compare models while taking the necessary precautions to prevent overfitting.

Hope this helps and good luck!

Answer 2

Adding to @AN6U5's respond.

From a purely theoretical perspective, this paper has show RNN are universal approximators. I haven't read the paper in details, so I don't know if the proof can be applied to LSTM as well, but I suspect so. The biggest problem with RNN in general (including LSTM) is that they are hard to train due to gradient exploration and gradient vanishing problem. The practical limit for LSTM seems to be around 200~ steps with standard gradient descent and random initialization. And as mentioned, in general for any deep learning model to work well you need a lot of data and heaps of tuning.

ARIMA model is more restricted. If your underlying system is too complex then it is simply impossible to get a good fit. But on the other hand, if you underlying model is simple enough, it is much more efficient than deep learning approach.

Answer 3

ARIMA models are linear and LSTM models are nonlinear. Some other parametric nonlinear time series models that statisticians have studied are Threshold Autoregressive Models (TAR) and Smooth Transition Autoregressive Models (STAR). The R package tsDyn implements these models.

I wonder how STAR models do vs. LSTM.

Answer 4

I’ve come to the same conclusion as yourself and others, traditional forecasting is still probably the most applicable and maybe reliable for time series of numeric values. There is some slight bleed in deep learning in discussion where time series for numeric values gets mixed into deep learning, where deep learning (currently) applies to modern challenges in pattern recognition for image, sound, clean text, or anomaly detection. I often have good results with VAR / VECM for daily transactional data, which could probably be applied to your signal processing use case.

Answer 5

As an extreme case, I had a chance to study on Forex (Foreign Exchange Rate) forecast and intensively compared performances of LSTM, windowed-MLP and ARIMA. As many articles say, Forex time series is close to the random walk series (it is completely non-stationary). None of these algorithms can predict next day's spot rate. For example, if there is no (or little) change, then it will maintain current value and it looks fit. However, if there is a sudden (substantial) change in tomorrow's spot rate, then it always fails to predict. The problem is that there is nothing to learn from the training data (say, past 10 years' spot rates history) whether it is large enough or not, because it is completely random (perhaps, due to exchange dealers' sentiments ...).

In fact, LSTM with just '1' lag, or ARIMA with (1,0,0) will perform the best since storing data with more than 1 time lag doesn't help at all. For this kind of time series, I would say, in terms of tracking performance, 1) ARIMA, 2) LSTM, 3) MLP. Last comment... MLP with macro/micro economic parameters (say, GDP difference, bond rates, crude oil price, dollar index, etc) as additional features didn't work at all also.

Answer 6

ARIMA gives Trend (or Regression) - can see its slope (that reflects speed of change dx/dy)...

LSTM gives MovingAverage - posessing curvature & slope in each moment - that can characterize not only speed but also acceleration changes to up or down

p.s.

in all times Lines in e.g. classical supply-demand charts in economics were considered to be Static Models (as all regressions), and Curves on such charts (instead of lines) reflects Dynamic Models (showing not only speed but also acceleration)

Answer 7

The library forecast in R can calculate all the parameters for Arima models automatically for you. No need to figure it out. You can ever provide it additional timeseries that could explain the residual error.

Answer 8

I think, you are misusing MachineLearning & Deep Learning when trying to predict tomorrow... - any statistical Approximations of the chaos can show just averaged Tendency & its borders (variance) according probability distributions, but not the obligations of the evaluated system to be determined exactly one way (because social systems and their derivatives are rather volatile) and of course not tomorrow -- all predictions according existing Tendency can be realized in time - TILL something will change this Tendency...

and recall once again: calculations in ProbabilityTheory cannot give prediction of exact event, but the range of possible events - each of them having their own probabilities, but still having these probabilities (even if being low) in a very volatile environment of social needs, habits, laws, the daily routine & regulations in each country & etc. influencing financial markets