# How do I perform multi-step forecasting on LSTM trained with multiple observations of a sequence?

I have a trained sequence-to-sequence LSTM that predicts a person's velocity trace (Δx, Δy) given their position (x, y) as they walk along a path. I trained the LSTM using 200 examples of different people walking along the path. Now, I would like to use multi-step forecasting to predict the future velocity of several new people after they have walked 1/3 of the path. Is this possible? If so, how can I implement this in Python or MATLAB?

Train = 200 sequence observations, 2 input features (x, y), time lengths of observations vary

Test = 5 partial sequence observations (1/3 of the sequence), I would like to predict the last 2/3 of the sequence for each person.

ConvLSTM, keras has an implementation, therefore you can go with Python itself.

If you want multiple outputs from the LSTM, you can have look at return_sequences and return_state feature in LSTM layers. Default values for them are None, But if you give True you can get multiple outputs for each timestep, and for everyone.

This excellent blog post helped me understand them.

One technique is optical flow [1], [2], which is popular with people doing modeling of action videos.

There is a github implementation of [3] here, which she calls ConvLSTM, and is coded in Lua.

[1] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid. Deepflow: Large displacement optical flow with deep matching. In Computer Vision (ICCV), 2013 IEEE International Conference on, pp. 1385–1392, Dec 2013. doi: 10.1109/ICCV.2013.175

[2] Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. CoRR, abs/1504.06852, 2015

[3] Patraucean, V., Handa, A., & Cipolla, R. (2015). Spatio-temporal video autoencoder with differentiable memory. arXiv preprint arXiv:1511.06309.