When modeling sequential data, RNNs are introduced as an improvement of MLP as they can model the time dependency between the inputs. It is said that feeding the last N data points in the sequence to a MLP does not work because it does not understand that the points come after each other in time. But is this not something that can be learned? Is it not possible the backprop can set the weights of the first layer such that the time dependency is modeled?


1 Answer 1


Yes, it can be learned. In fact many time series problems are solved with non-RNN/LSTM/1D-CNN architectures due to their simplicity.

The reason the more exotic architectures are used are to increase performance. Instead of getting the classic DNN to learn the time relationship implicitly, we can inject our domain knowledge into the problem by choosing a more performant architecture.

It is similar to using classic vs Convolutional NNs, both work, one more directly solves the problem.

The way the increase in performance works, is that you need many less parameters. Look at how RNN cells are structured, and it will make sense that you only need the input, and hidden state from previous RNN cell.

The next reason pointed out by https://stats.stackexchange.com/a/409941/257148 is RNNs can take sequences of arbitrary length.

You can build performant models with most architectures, I've found the difference is the level of feature engineering required. With non-exotic architectures, you need to feed in a lot of differenceing and lag features into your model. This can get expensive, but gives you a lot of control. This is good on small problems, or if you have a lot of compute - as the hyper-parameter search and training times take a long time. I've personally won time-series hackathons using both feature-engineering + classic architecture, and RNN/1-D CNN/LSTMs.

  • $\begingroup$ Would adding a layer of the same breadth as the input to the MLP and enforcing the weights of this layer to be an upper triangular matrix be similar to a RNN? $\endgroup$ Sep 4, 2022 at 12:34
  • $\begingroup$ I'm not sure sorry, I'd ask - why not just use a RNN instead of trying to replicate it $\endgroup$
    – GooJ
    Sep 4, 2022 at 17:42
  • $\begingroup$ Just to make sure I have understood. $\endgroup$ Sep 5, 2022 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.