Input sequence ordering for LSTM network

Question

When training a LSTM network with time series data, I guess the order in which this data is fed matters, my question is how should this ordering be...

Let's take a time-series vector which will be the input for the LSTM:

\[ X = [x_0, x_{\small(-1\small)}, x_{\small(-2\small)},\ldots, x_{\small(-N\small)}]\]

with negative indexing indicating past values.

Which vector should be fed (theoretical case, not related to any API)?:

• \[ X = [x_0, x_{\small(-1\small)}, x_{\small(-2\small)},\ldots, x_{\small(-N\small)}]\]

or

• \[ X^r = [x_{\small(-N\small)},\ldots, x_{\small(-2\small)}, x_{\small(-1\small)}, x_0]\]

More precisely, what of these two orderings should be used in TensorFlow LSTM-Cells?

Answer 1

In order to make this decision, you have to think about what you want the representation to be passed to the next layer (or network output) to represent.

If you want the representation at (after) $t = 0$ to be passed, you should pass the arrays in the $X = [x_{-n} .. x_0]$ order. The LSTM cell will form a representation of the sequence, then, at $t = 0$.

In the reverse case, the representation will indicate the state at (before) $t = -n$

That's not to say that this is always a simple question for LSTM design- frequently in some domains such as NLP (text modeling), a bidirectional LSTM is used. This, basically, means both representations are implemented and used.