Last cell in recurrent network always the most accurate

Question

I am using a recurrent network for time series forecasting.

The prediction from the last cell in the network always seems to be the most accurate. For example if I have 20 cells (so my input samples are of length 20), the prediction from the latter cells (e.g. cells 11-20) seem better than the predictions from the early cells (e.g. cells 1-10).

Is this normal?

Lets assume my network only has 3 cells. At present I would input data x(t=0, t+1, t+2) to obtain predictions for t=0, t+1, t+2.Is this the wrong approach? If I want good predictions at all timesteps, do I need to using a "moving window" i.e. should I use a rolling sequence of inputs:

For prediction at t=0, input x(t-2, t-1, t=0)
For prediction at t=1, input x(t-1, t=0, t=1)
For prediction at t=2, input x(t=0, t+1, t+2)

Answer 1

I am using a recurrent network for time series forecasting.

Interesting would be especially the type of the network you chose. How many layers? How many neurons per layer?

The prediction from the last cell in the network always seems to be the most accurate.

I assume by cells you refer to your samples. Always, if you learn in datastreams (online), the network in later iterations has seen more data than the network in early iterations. Thus, with having seen more samples, your predictions become more accurate. Of course, this assumes that you do online learning on your datastream.

At present I would input data x(t=0, t+1, t+2) to obtain predictions for t=0, t+1, t+2.Is this the wrong approach?

This depends on how your RNN is set up. A Jordan Network for example has context neurons, which should do exactly what you propose. The reason for recurrency in the network is typically that you want to simulate short term memory - meaning, in a mathematical view, you want to feed information from previous iterations into the front of the network. Thus, since this feature should be taken care of in your RNN, I do not think it is good to extend your input, since you introduce additional, potentially useless weights that have to be learned by the network.