What if we input sequence data to feedforward network?

Question

One main advantage of RNN is the ability to take input of variable length like the case of sequences. However, what if we neglected this case and assumed some applications that may accept some fixed window of input. For example, the weather prediction. We may input the temperature of the last 300 days for example to allow the network to predict the new temperature. We may fix this window of 300 all the time so that we have a fixed-length input.

My question now is what if we neglect the RNN in this case and input this window of 300 days into a feedforward network with 300 input dimensions and one output that is doing regression for it? What will be the output as compared to RNN? In other words, why I should use RNN in this case?

Answer 1

It all depends on the nature of the data in relation to the labels.

For instance, if all that you need to appropriately classify the input sequence is to know the values at certain fixed points, then a mere multi-layer perceptron (MLP) could do.

However, if in order to properly classify it was needed to take a look at the trends, maybe the MLP would not behave so well (this would depend on the actual data, of course).

If could be the case that the labels depend on some local patterns in the daily values. In that case, maybe a 1D convolutional network could do well, because local pattern detection is precisely their inductive bias.

Recurrent networks' inductive bias is inherently sequential, and therefore fair well when the prediction can be obtained when looking value after value. Of course, people normally use LSTMs or GRUs instead of vanilla RNNs due to the vanishing gradient problem.

Finally, self-attention networks, which are feedforward, are currently the state of the art is natural language processing. These graph neural networks in disguise can obtain in general better text representations than LSTMs.

So, summing up: it is perfectly possible to use a feedforward network on sequential data.

Answer 2

A finite impulse response (FIR) RNN can be expressed as a directed acyclic graph (DAG), hence it can be represented as a FFNN. So, you could theoretically make an equivalent FFNN to an RNN in this situation.

In terms of performance between non-equivalent models, an RNN would probably be better because, as they are inherently designed for sequential data. In the case of weather stat prediction, the temporal nature of the sequence elements would need to be learned by a FFNN, whereas an RNN would have in-built knowledge of such relationships. The RNN would also be able to more easily take advantage of "memory" via the common gating mechanisms as seen in LSTMs and the like.