66
$\begingroup$

The problem that I am dealing with is predicting time series values. I am looking at one time series at a time and based on for example 15% of the input data, I would like to predict its future values. So far I have come across two models:

  • LSTM (long short term memory; a class of recurrent neural networks)
  • ARIMA

I have tried both and read some articles on them. Now I am trying to get a better sense on how to compare the two. What I have found so far:

  1. LSTM works better if we are dealing with huge amount of data and enough training data is available, while ARIMA is better for smaller datasets (is this correct?)
  2. ARIMA requires a series of parameters (p,q,d) which must be calculated based on data, while LSTM does not require setting such parameters. However, there are some hyperparameters we need to tune for LSTM.
  3. EDIT: One major difference between the two that I noticed while reading a great article here, is that ARIMA could only perform well on stationary time series (where there is no seasonality, trend and etc.) and you need to take care of that if want to use ARIMA

Other than the above-mentioned properties, I could not find any other points or facts which could help me toward selecting the best model. I would be really grateful if someone could help me finding articles, papers or other stuff (had no luck so far, only some general opinions here and there and nothing based on experiments.)

I have to mention that originally I am dealing with streaming data, however for now I am using NAB datasets which includes 50 datasets with the maximum size of 20k data points.

$\endgroup$
26
$\begingroup$

Statement 1 is correct, statement 2 is correct, but requires elaboration, and statement 3 is incorrect for seasonal ARIMA:

The following might point you in the right direction but hopefully you'll get a few more answers with more depth in the arena of LSTM.

You mention that you have tried both algorithms and that you are simply trying to figure out which one is better, which leads me to think you may be having more trouble with the data science process and cross validation than with the specifics of the models.

Time series in general:

Time series, in general, are difficult to forecast. If they were easy to forecast then all data scientists would be wealthy, having accurately forecast the value of all of the stocks. The reality is that hedge funds, on average, do not outperform the market and that time series forecasting is typically very poor and applies only to very short durations. The main problems are that there is a lot of noise, there are many hidden influences, models are overly simplistic, influencers do not behave as we think they should, the interplay between linearity and nonlinearity is subtle and confusing, ... ad infinitum.

ARIMA

You are incorrect in your assessment that ARIMA requires stationary time series to forecast on. Non-seasonal ARIMA has three input values to help control for smoothing, stationarity, and forecasting ARIMA(p,d,q), where:

  • p is the number of autoregressive terms,
  • d is the number of nonseasonal differences needed for stationarity, and
  • q is the number of lagged forecast errors in the prediction equation.

By contrast seasonal ARIMA has six input values ARIMA(p,d,q,P,D,Q), where:

  • P is the number of seasonal autoregressive terms,
  • D is the number of seasonal differences, and
  • Q is the number of seasonal moving-average terms.

Subject to the qualifying statements above, I suggest playing with seasonal ARIMA to get a feel for the intricacies involved in smoothing, de-seasoning, de-trending, de-noiseing, and forecasting.

LSTM

I don't know enough about LSTM to add much here. I will add that red flags tend to be raised when someone begins at data science exercise with deep learning. I suggest learning as much as you can using ARIMA and then applying some of your ARIMA expertise to help you learn LSTM. Neural networks can be a very powerful tool, but they:

  • can take a long time to run,
  • often require more data to train than other models, and
  • have lots of input parameters to tune.

Cross validation and comparing models:

Time series are fun in that all training data can usually be turned into supervised learning training sets. Once can simply take a time series and roll back time. That is... pick a point in time and pretend that you don't have any additional data, then produce a forecast and see how well you did. You can march through the time series doing this $n$ times in order to get an assessment of the performance of your model and to compare models while taking the necessary precautions to prevent overfitting.

Hope this helps and good luck!

$\endgroup$
  • $\begingroup$ Thank you for you response. Your notes on ARIMA are helpful and I try to keep them in mind. In case of cross validation and comparing the two models, for the current datasets, using variety of parameters I can tell which algorithm is performing better and so on. The problem is that non of those datasets could represent the real data (streaming data) and that's why I figured it would be better to ask others in the area of data science and machine learning to share their thoughts and experiences. $\endgroup$ – ahajib Jul 25 '16 at 13:48
  • $\begingroup$ But if you have streaming data then you can turn this into testing data by saving the stream. The nice part about time series is that you can always create a testing data set out of any data set. You just need to save some of the data and then roll back time. $\endgroup$ – AN6U5 Jul 25 '16 at 13:58
  • $\begingroup$ I know this is kind of aged but I have to disagree here.The d, D parameters are for differencing like @AN6U5 says, but the reason to do these differences in the first place is to enforce stationarity on the problem. In fact one interpretation of the ARIMA model is that is just linear regression with terms to enforce stationarity on the time-series. $\endgroup$ – Ryan Nov 26 '18 at 16:31
6
$\begingroup$

Adding to @AN6U5's respond.

From a purely theoretical perspective, this paper has show RNN are universal approximators. I haven't read the paper in details, so I don't know if the proof can be applied to LSTM as well, but I suspect so. The biggest problem with RNN in general (including LSTM) is that they are hard to train due to gradient exploration and gradient vanishing problem. The practical limit for LSTM seems to be around 200~ steps with standard gradient descent and random initialization. And as mentioned, in general for any deep learning model to work well you need a lot of data and heaps of tuning.

ARIMA model is more restricted. If your underlying system is too complex then it is simply impossible to get a good fit. But on the other hand, if you underlying model is simple enough, it is much more efficient than deep learning approach.

$\endgroup$
3
$\begingroup$

I’ve come to the same conclusion as yourself and others, traditional forecasting is still probably the most applicable and maybe reliable for time series of numeric values. There is some slight bleed in deep learning in discussion where time series for numeric values gets mixed into deep learning, where deep learning (currently) applies to modern challenges in pattern recognition for image, sound, clean text, or anomaly detection. I often have good results with VAR / VECM for daily transactional data, which could probably be applied to your signal processing use case.

$\endgroup$
2
$\begingroup$

ARIMA models are linear and LSTM models are nonlinear. Some other parametric nonlinear time series models that statisticians have studied are Threshold Autoregressive Models (TAR) and Smooth Transition Autoregressive Models (STAR). The R package tsDyn implements these models.

I wonder how STAR models do vs. LSTM.

$\endgroup$
1
$\begingroup$

As an extreme case, I had a chance to study on Forex (Foreign Exchange Rate) forecast and intensively compared performances of LSTM, windowed-MLP and ARIMA. As many articles say, Forex time series is close to the random walk series (it is completely non-stationary). None of these algorithms can predict next day's spot rate. For example, if there is no (or little) change, then it will maintain current value and it looks fit. However, if there is a sudden (substantial) change in tomorrow's spot rate, then it always fails to predict. The problem is that there is nothing to learn from the training data (say, past 10 years' spot rates history) whether it is large enough or not, because it is completely random (perhaps, due to exchange dealers' sentiments ...).

In fact, LSTM with just '1' lag, or ARIMA with (1,0,0) will perform the best since storing data with more than 1 time lag doesn't help at all. For this kind of time series, I would say, in terms of tracking performance, 1) ARIMA, 2) LSTM, 3) MLP. Last comment... MLP with macro/micro economic parameters (say, GDP difference, bond rates, crude oil price, dollar index, etc) as additional features didn't work at all also.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.