1
$\begingroup$

Setup: We have sequence of events that are not evenly spaced (not a time series). Length of the sequence is constant.

Goal: Predict class of the event that is most probable to follow this sequence.

Background: I know that RNN would probably a good fit for this task, but at the same time I wonder whether parameters sharing in our U,W,V matrices actually hurt accuracy ( even though training process is cheaper). Let's say we are ok to spent more time(and data) for training and don't want to compromise accuracy.

Question: Is it true that by using regular MLP we can achieve better or at least same performance if we just combine/flatten all features from those sequence events and pass them alltogether as an input? I believe model should still be able to learn interactions between features(that represent different events in a sequence) but not sure how good it will be at it and if not then why?

Improve this question
$\endgroup$
0
$\begingroup$

Conceptually, the sound of dropping a metal chain on the floor is different from the sound of dropping the separated links of that chain.

Feedforward NN

In a feed forward neural network all of sequential features would be consumed independently:

$f(x)=WX+b=w_1x_1+..+w_nx_n+b$

This is all good, as far as you’re ready to sacrifice the step-wise dependency between x1, x2 etc.

Recurrent NN

To be able to utilise the temporal or sequential signal in your dataset you need a method that “chains” each feature with its past/future state, right?

We "connect" these sequential events with the alleged "hidden state":

$a_n = f(W_n, a_{n-1}, x_n)$

By exploding the above equation you can see how past information is accumulated in the subspace of the hidden state:

$a_n = f(W_n, a_{n-1}, x_n) = f(W_n, f(W_{n-1}, a_{n-2}, x_{n-1}), x_n)$, since $ a_{n-1}=f(W_n, a_{n-2}, x_n)$.

You may be able to see some issues with the above especially when it comes to sequences of larger length, usually tackled with LSTM and attention pooling architectures, but this is a different discussion!

Hope it helps.

Improve this answer
$\endgroup$
3
  • $\begingroup$ thank you! Let me try to elaborate on the source of my confusion. According to wikipedia FFNN can learn functions like XOR(x1,x2) - interactions between inputs. So hidden neurons would have access to all inputs x1, x2,etc (which represent features from different time steps) at once (which even feels like bi-direct. RNN). RNN on the other hand uses Markov assumption and at every time step takes into account only current input and previous output (which contains by recursion all previous inputs). Feels like both should be able to learn dependencies but what is the difference? $\endgroup$
    – Deil
    Apr 30 '21 at 20:28
  • $\begingroup$ Sorry, are you saying that XOR equals sequential feature dependency? Surely a ffNN allows feature interaction modelling but not sure how it is argued that it is the equivalent of imposing a step-wise feature dependency on the underlying data distribution. $\endgroup$
    – hH1sG0n3
    May 1 '21 at 10:32
  • $\begingroup$ sorry for confusion, I'm not saying that. I was trying to say that ffNN allows feature interaction modelling. For a given task we can pretend there is no notion of time and just try to predict event C given A, B. Let's say we have lots of training examples A, B so that we perfectly can predict C(even without imposing step-wise dependency). So I'm trying to get a better intuition on when would we need to use RNN( like when it is more than k time steps it will require too many features for ffNN case and curse of dimensionality will motivate us to use RNN). There must be a decision line $\endgroup$
    – Deil
    May 2 '21 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.