LSTM: many to one and many to many in time-series prediction

Question

I am trying to predict the trajectory of an object over time using LSTM. I have three different configurations of training and predicting values in my mind and I would like to know what the best solution to this problem might be (I would also appreciate insights regarding these approaches).

1) Many to one (loss is the MSE of a single value)

• The $input$ is a sequence of $n$ values, the output is the prediction of the single value at position $n+1$.
• The loss function is the MSE of the predicted value and its real value (so, corresponding to the value in position $n+1$).
• During the online test, a sequence of $n$ values predict one value ($n+1$), and this value is concatenated to the previous sequence in order to predict the next value ($n+2$) etc.. This way, a whole trajectory of $n + t$ values is calculated.

2) Many to one (loss is MSE of multiple values)

• The $input$ is a sequence of $n$ values, the output is the prediction of the single value at position $n+1$.
• To compute the loss function, the same strategy used before for online test is applied. LSTM predicts one value, this value is concatenated and used to predict the successive value $t$ times. The loss is the MSE of all the predicted values in the trajectory and their real values. Backpropagation is only done when the whole trajectory has been predicted.
• Online testing is equal to the previous situation.

3) Many to many

• The $input$ is a sequence of $n$ values, the output is the prediction of $m$ consecutive values.
• The loss function is the MSE of the $m$ predictions and their corresponding ground truth.

Answer 1

The choice is mostly about your specific task: what do you need/want to do?

Many-to-one (single values) models have lower error, on average, since the quality of outputs decreases the more further in time you're trying to predict. Many-to-one (multiple values) sometimes is required by the task though.

An alternative could be to employ a Many-to-one (single values) as a (multiple values) version: you train a model as (single), then you use it iteratively to predict multiple steps. I personally experimented with all these architectures, and I have to say this doesn't always improves performance.

Based on my experience, Many-to-many models have better performances. For example, I had to implement a very large time series forecasting model (with 2 steps ahead prediction). The best model was returning the same input sequence, but shifted forward in time of two steps. It appeared that the model was better at keeping the predicted values more coherent with previous input values. It was a seq2seq RNN with LSTM layers.

---

EDIT:

The Loss doesn't strictly depend on the version, each of the Losses discussed could be applied to any of the architectures mentioned. A problem for multiple outputs would be that your model assigns the same importance to all the steps in prediction. This is something you can fix with a custom MSE Loss, in which predictions far away in the future get discounted by some factor in the 0-1 range. In that way your model would attribute greater importance to short-range accuracy. Let me know if that's helpful.