# Sequence Batching in RNNs

I'm wondering why sequence batching in RNNs's target value loops back (I'm not sure what you call it), but let's take for example:

We want to learn a sequence of numbers (our input) from 1 to 16:

$$\begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 13 & 14 & 15 & 16 \end{bmatrix}$$

Batches: 2, Sequence Length: 4

First, we can divide the data to 2 batches:

$$\begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8\\ 9 & 10 & 11 & 12 & 13 & 14 & 15 & 16 \end{bmatrix}$$

Then we can divide this into mini batches:

$$\begin{bmatrix} 1 & 2 & 3 & 4\\ 9 & 10 & 11 & 12 \end{bmatrix}$$ $$\begin{bmatrix} 5 & 6 & 7 & 8\\ 13 & 14 & 15 & 16 \end{bmatrix}$$

Then we need to create targets for the inputs, and intuitively we want to to targets to be the next value of the input, so:

$$\begin{bmatrix} 2 & 3 & 4 & 5\\ 10 & 11 & 12 & 13 \end{bmatrix}$$

However, this is not what I usually see, instead I see the last value in a mini batch is swapped with the first value:

$$\begin{bmatrix} 2 & 3 & 4 & 1\\ 10 & 11 & 12 & 9 \end{bmatrix}$$

So what is the intuition in doing so?

Since if we want to learn the sequence of 1, 2, 3, 4, but 1 was given as the target for the value 3, so 4 was not learnt but instead of 1.

• Where did you read that?
– Emre
Jun 8 '17 at 21:06
• @Emre, I'm working on a tutorial on RNNs, and it looks like data are given that way on purpose with a comment along the lines of: It's done this way usually, and it doesn't effect performance. But I found it to be unintuitive. Jun 8 '17 at 22:30
• There is no such rule. The burden of proof is on them.
– Emre
Jun 8 '17 at 22:33
• @Emre, I thought so, since we want to learn how to count up, and going down in the middle doesn't make much sense. Do you want to give it as an answer? Jun 8 '17 at 22:36