1
$\begingroup$

I'm wondering why sequence batching in RNNs's target value loops back (I'm not sure what you call it), but let's take for example:

We want to learn a sequence of numbers (our input) from 1 to 16:

$$ \begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 13 & 14 & 15 & 16 \end{bmatrix} $$

Batches: 2, Sequence Length: 4

First, we can divide the data to 2 batches:

$$ \begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8\\ 9 & 10 & 11 & 12 & 13 & 14 & 15 & 16 \end{bmatrix} $$

Then we can divide this into mini batches:

$$ \begin{bmatrix} 1 & 2 & 3 & 4\\ 9 & 10 & 11 & 12 \end{bmatrix} $$ $$ \begin{bmatrix} 5 & 6 & 7 & 8\\ 13 & 14 & 15 & 16 \end{bmatrix} $$

Then we need to create targets for the inputs, and intuitively we want to to targets to be the next value of the input, so:

$$ \begin{bmatrix} 2 & 3 & 4 & 5\\ 10 & 11 & 12 & 13 \end{bmatrix} $$

However, this is not what I usually see, instead I see the last value in a mini batch is swapped with the first value:

$$ \begin{bmatrix} 2 & 3 & 4 & 1\\ 10 & 11 & 12 & 9 \end{bmatrix} $$

So what is the intuition in doing so?

Since if we want to learn the sequence of 1, 2, 3, 4, but 1 was given as the target for the value 3, so 4 was not learnt but instead of 1.

$\endgroup$
  • 1
    $\begingroup$ Where did you read that? $\endgroup$ – Emre Jun 8 '17 at 21:06
  • $\begingroup$ @Emre, I'm working on a tutorial on RNNs, and it looks like data are given that way on purpose with a comment along the lines of: It's done this way usually, and it doesn't effect performance. But I found it to be unintuitive. $\endgroup$ – user1157751 Jun 8 '17 at 22:30
  • $\begingroup$ There is no such rule. The burden of proof is on them. $\endgroup$ – Emre Jun 8 '17 at 22:33
  • $\begingroup$ @Emre, I thought so, since we want to learn how to count up, and going down in the middle doesn't make much sense. Do you want to give it as an answer? $\endgroup$ – user1157751 Jun 8 '17 at 22:36
2
$\begingroup$

It makes no sense to re-order inputs in the general case because the order might matter. In your example it does not; you can shuffle the columns as long as the corresponding outputs remain the same.

I've seen the input reversed, which is a less arbitrary transformation than the one you cite, to improve prediction in sequence-to-sequence models, though that's not set in stone, either.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.