Can MLP model sequential data?

Question

When modeling sequential data, RNNs are introduced as an improvement of MLP as they can model the time dependency between the inputs. It is said that feeding the last N data points in the sequence to a MLP does not work because it does not understand that the points come after each other in time. But is this not something that can be learned? Is it not possible the backprop can set the weights of the first layer such that the time dependency is modeled?

Answer 1

Yes, it can be learned. In fact many time series problems are solved with non-RNN/LSTM/1D-CNN architectures due to their simplicity.

The reason the more exotic architectures are used are to increase performance. Instead of getting the classic DNN to learn the time relationship implicitly, we can inject our domain knowledge into the problem by choosing a more performant architecture.

It is similar to using classic vs Convolutional NNs, both work, one more directly solves the problem.

The way the increase in performance works, is that you need many less parameters. Look at how RNN cells are structured, and it will make sense that you only need the input, and hidden state from previous RNN cell.

The next reason pointed out by https://stats.stackexchange.com/a/409941/257148 is RNNs can take sequences of arbitrary length.

You can build performant models with most architectures, I've found the difference is the level of feature engineering required. With non-exotic architectures, you need to feed in a lot of differenceing and lag features into your model. This can get expensive, but gives you a lot of control. This is good on small problems, or if you have a lot of compute - as the hyper-parameter search and training times take a long time. I've personally won time-series hackathons using both feature-engineering + classic architecture, and RNN/1-D CNN/LSTMs.