2
$\begingroup$

Suppose I have a single sequence of $x_1, x_2, ..., x_n$ and corresponding labels $y_1, y_2, ..., y_n$.

An example would be a person makes website visits $x_i$ and the label $y_i$ tells us if there was a purchase in any of the previous visits.

Is there a difference between training multiple sequence to one or a single sequence to sequence?

Multiple sequence to one

This approach gives us $n$ data points. Additionally it seems that the model will see $x_1$ n times, while $x_n$ will be only fed in once.

$[x_1] , [y_1]$

$[x_1, x_2] , [y_2]$

...

$[x_1, x_2, ..., x_n], [y_n]$

Single Sequence to Sequence

This approach gives us just 1 data point.

$[x_1, x_2, ..., x_n], [y_1, y_2, ..., y_n]$, with no repetition in data.

Do these yield the same outcome in RNN (e.g. LSTM) training? It seems that when your data is imbalanced, you would greatly benefit from the 1st approach, since you get more observations.

My understanding is that, since even a single data point is technically a valid data point in my case, I should be using the multiple sequence to one approach. But in cases of NLP (where the first word alone has no context), obviously we need the whole sequence.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
+50
$\begingroup$

As you are talking about RNN, I'm going to change the parameters of your question a little bit.

The multi sequence to one will be:

$[x_1] , [y_1]$

$[x_1, x_2, y_1] , [y_2]$

...

$[x_1, x_2, ..., x_n, y_{(n-1)}], [y_n]$

as the previous state will always be used.

This is typically a Markov process with a state containing a n input vector. It has known properties and that's what is used by RNN to be able to create a sequence one element by one element.

On the case of one sequence to one sequence, there is no such state, and we don't have the concept of one word following the other. It's basically n values in the input and n value as the output. But there is no constraint about "one after the other" in such a pseudo sequence anymore, so the outputs could be scrambled and it would still be a sequence to sequence output.

Now, for NLP, the fact that you don't have a state before time 0 is not a problem. You can discard the first n-1 elements even and keep the Markov property of the system.

This is exactly what I'm doing in chapter 8 in Building Machine Learning with Python and it works well if you start retrieving words when the system is in a proper state.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for your answer. You mention that there is no state in sequence to sequence, but to make a prediction of $y_2$, we do pass the state from the first cell (which is $y_1$). We pass the state $y_{i-1}$ to make a prediction for $y_i$ alongside using $x_i$ input. So I don't quite understand when you say there is no such state passed?Please refer to colah.github.io/posts/2015-08-Understanding-LSTMs/img/… , I'm talking about the horizontal arrow which passes the state. $\endgroup$
    – GRS
    Dec 20, 2018 at 11:02
  • $\begingroup$ To me, it clearly defines the order (one after another), since we define which sequences are fed in first. Unless I'm misunderstanding what is actually happening. $\endgroup$
    – GRS
    Dec 20, 2018 at 11:04
  • $\begingroup$ OH, I see. For me such a case is just a rewording of the multiple to one, but written in a different way. We keep all the important bits from Markov chain and we do process the LSTM n times to get a full n answer. $\endgroup$
    – Matthieu Brucher
    Dec 20, 2018 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.