2
$\begingroup$

One main advantage of RNN is the ability to take input of variable length like the case of sequences. However, what if we neglected this case and assumed some applications that may accept some fixed window of input. For example, the weather prediction. We may input the temperature of the last 300 days for example to allow the network to predict the new temperature. We may fix this window of 300 all the time so that we have a fixed-length input.

My question now is what if we neglect the RNN in this case and input this window of 300 days into a feedforward network with 300 input dimensions and one output that is doing regression for it? What will be the output as compared to RNN? In other words, why I should use RNN in this case?

$\endgroup$
1
  • 1
    $\begingroup$ Please, consider providing feedback about the answers to your question, and upvoting and marking an answer as correct, if appropriate. $\endgroup$ – noe Oct 15 '20 at 15:18
0
$\begingroup$

A finite impulse response (FIR) RNN can be expressed as a directed acyclic graph (DAG), hence it can be represented as a FFNN. So, you could theoretically make an equivalent FFNN to an RNN in this situation.

In terms of performance between non-equivalent models, an RNN would probably be better because, as they are inherently designed for sequential data. In the case of weather stat prediction, the temporal nature of the sequence elements would need to be learned by a FFNN, whereas an RNN would have in-built knowledge of such relationships. The RNN would also be able to more easily take advantage of "memory" via the common gating mechanisms as seen in LSTMs and the like.

$\endgroup$
0
$\begingroup$

It all depends on the nature of the data in relation to the labels.

For instance, if all that you need to appropriately classify the input sequence is to know the values at certain fixed points, then a mere multi-layer perceptron (MLP) could do.

However, if in order to properly classify it was needed to take a look at the trends, maybe the MLP would not behave so well (this would depend on the actual data, of course).

If could be the case that the labels depend on some local patterns in the daily values. In that case, maybe a 1D convolutional network could do well, because local pattern detection is precisely their inductive bias.

Recurrent networks' inductive bias is inherently sequential, and therefore fair well when the prediction can be obtained when looking value after value. Of course, people normally use LSTMs or GRUs instead of vanilla RNNs due to the vanishing gradient problem.

Finally, self-attention networks, which are feedforward, are currently the state of the art is natural language processing. These graph neural networks in disguise can obtain in general better text representations than LSTMs.

So, summing up: it is perfectly possible to use a feedforward network on sequential data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.