# Sequence Batching in RNNs

I'm wondering why sequence batching in RNNs's target value loops back (I'm not sure what you call it), but let's take for example:

We want to learn a sequence of numbers (our input) from 1 to 16:

$$\begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 13 & 14 & 15 & 16 \end{bmatrix}$$

Batches: 2, Sequence Length: 4

First, we can divide the data to 2 batches:

$$\begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8\\ 9 & 10 & 11 & 12 & 13 & 14 & 15 & 16 \end{bmatrix}$$

Then we can divide this into mini batches:

$$\begin{bmatrix} 1 & 2 & 3 & 4\\ 9 & 10 & 11 & 12 \end{bmatrix}$$ $$\begin{bmatrix} 5 & 6 & 7 & 8\\ 13 & 14 & 15 & 16 \end{bmatrix}$$

Then we need to create targets for the inputs, and intuitively we want to to targets to be the next value of the input, so:

$$\begin{bmatrix} 2 & 3 & 4 & 5\\ 10 & 11 & 12 & 13 \end{bmatrix}$$

However, this is not what I usually see, instead I see the last value in a mini batch is swapped with the first value:

$$\begin{bmatrix} 2 & 3 & 4 & 1\\ 10 & 11 & 12 & 9 \end{bmatrix}$$

So what is the intuition in doing so?

Since if we want to learn the sequence of 1, 2, 3, 4, but 1 was given as the target for the value 3, so 4 was not learnt but instead of 1.

• Where did you read that? – Emre Jun 8 '17 at 21:06
• @Emre, I'm working on a tutorial on RNNs, and it looks like data are given that way on purpose with a comment along the lines of: It's done this way usually, and it doesn't effect performance. But I found it to be unintuitive. – user1157751 Jun 8 '17 at 22:30
• There is no such rule. The burden of proof is on them. – Emre Jun 8 '17 at 22:33
• @Emre, I thought so, since we want to learn how to count up, and going down in the middle doesn't make much sense. Do you want to give it as an answer? – user1157751 Jun 8 '17 at 22:36