Why are RNN/LSTM preferred in time series analysis and not other NN?

Question

I had recently a great discussion about the advantages of RNN/LSTM in time series analysis in comparison to other Neural Networks like MLP or CNN.

The other side said, that:

1. The NN just have to be deep enough to model the time connections

2. RNNs are just used, because they are faster

3. With RNN the older (e.g. t-20) timepoints are not that relevant in comparison to the newer timepoints (e.g. t-1), because the data from older timepoints have to go through all neurons until the weight is updated in this neuron.

4. It is easier to understand the weight matrix of normal NN in comparison to RNN to understand partly the behavior of the NN

My answer to that was:

1. Everyone suggests using the LSTMs for times series analysis

2. It can consider the time better in comparison to a sliding window approach especially if the length of the sequence is quite long

3. To 3. That happens just if you use vanilla RNN instead of LSTM, because of the gradient explosion

So I know that I did not answer him completely (and maybe not correctly) and I am now really interested in the real reasons why LSTMS are suggested the most of the times. Maybe he is right? What do you think?

• Can a normal NN model the time connections the same way like a RNN/LSTM does when it is just deep enough?
• Does an RNN need more or less data in comparison to a NN to get the
  same/ better results?
• Are there time series where normal NN or RNN/LSTM perform better?
• Is it time data depended which Model will perform the best or are there some guidelines?
• Can the NN behavior be understood better than the RNN behavior?

Answer 1

I'll try to provide some insight which will hopefully help.

• Can a normal NN model the time connections the same way like a RNN/LSTM does when it is just deep enough?

Every neural net gets better in theory if it gets deeper. For a regular NN to model time connections properly, you could use the last n time steps as your input and the n+1th time step as your target. This will generate your training set and depending on your data, you could be able to model your time series fairly efficiently.

All of the most obvious pitfalls of this approach are actually addressed by RNNs/LSTM.

• Does an RNN need more or less data in comparison to a NN to get the same/ better results?

Difficult question, which would probably require some empirical results to check that theory. Also, with neural nets, sometimes, it's more about how fast it trains rather than how much training data it has that will make the biggest difference in performance.

To me, the main difference is that your regular NN will need a fixed-size input, whereas your RNN will be able to learn with input "up to" a certain size, which can be a big advantage to model the entire time series well.

• Are there time series where normal NN or RNN/LSTM perform better?

Again, this is a difficult question as it will depend on the data, the architecture of the networks, the training time etc.

I haven't done empirical research on this, but I'd say that your best guess would be time series that are rather short (less than 100 time steps for instance). If the time series is short, you might not need to model such an intricate relationship through time, which a regular NN could perhaps do as well as an RNN.

• Is it time data depended which Model will perform the best or are there some guidelines?

As explained above, every learning exercise will highly depend on the architecture and hyperparameter used, even the initialization of your weights, whether you use pre-training or not, and of course, your data. Again, I think that the shortest the time series, the more competitive regular NNs could be. But again, this is merely intuition-based and hasn't been checked thoroughly.

- Can the NN behavior be understood better than the RNN behavior?

It all depends on what you mean by understanding. RNN have more weights than NN so ultimately, there will be more things to analyze and eventually understand. But, with more data also comes more information, so perhaps there is more information to be gained from an RNN than a simple NN, even if it's a deep one.

Plus, sometimes, based on the initialization of the weights and other parameters, the interpretation of the model could vary, even if it's trained on the same data.

Answer 2

RNNs aren't necessarily faster. It's hard to compare different architectures, but in a text-mining application I'm doing, my RNN takes 3-5x longer than my CNN to train and score.

Long-distance relationships will take a lot of data to learn with any model. And it's not clear to me that paying more attention to more recent data is a bad thing. Systems change over time and so paying a lot of attention to ancient data probably isn't useful.

What do you mean by "understand": Comprehended in the abstract, visualize, probe about specific predictions?

For a cool RNN visualization (text): https://arxiv.org/abs/1506.02078 .

The choice isn't "LSTMs are all hype" versus "LSTMs ROCK". For example, in text use (categorical time series of words or characters), CNNs are better at some things, while LSTMs are better at others.

And theoretically, RNNs (including LSTMs, of course) can process varying-length series straightforwardly, while CNNs or other NNs need some work. (In practice -- for example if you're using tensorflow as your underlying engine -- you won't be able to handle absolutely any length of series with RNNs, but the principle is there.)