Philosophical question on redundancy

Question

Suppose I implement a supervised learning version of LSTM similar to this. Namely, I have these univariate time series data:

t Y
1 101
2 105
3 108
4 104
5 110
6 112
7 119
8 111
9 113
10 115

So Y at time 1 is 101, Y at time 2 is 105, etc.

The goal is to predict Y at time t+1 given Y at time t and t-1. That is, I can rearrange this data set to look like supervised learning:

Y_(t-1) Y_t Y_(t+1)
101 105 108
105 108 104
108 104 110
104 110 112
110 112 119
112 119 111
119 111 113
111 113 115

Notice here that each Y value from t=3 to t=10 (max) gets predicted only once.

Now say I partitioned my data set into overlapping minibatches of size 5. We'll consider only the first two minibatches for simplicity:

t Y
1 101
2 105
3 108
4 104
5 110

2 105
3 108
4 104
5 110
6 112

I do the same trick of converting this to a supervised learning problem for each minibatch:

Y_(t-1) Y_t Y_(t+1)
101 105 108
105 108 104
108 104 110

Y_(t-1) Y_t Y_(t+1)
105 108 104
108 104 110
104 110 112

Notice: now two of the predictions from the first minibatch are repeated in the second minibatch.

I know that if I had wanted to have only one prediction per Y value, then I should have converted the entire dataset to supervised first and then created minibatches.

Questions:

Is this alternative approach incorrect? If so, why? If not, should I average the predictions, or is there another accepted way of combining the predictions from several time steps like this?

EDIT: I should mention that the learned parameters of the model are different for each position, even though the data points are the same. For example, the parameters that would be used to predict Y_(t+1) = 104 in minibatch 1 are different from the parameters that would be used to predict Y_(t+1) = 104 in minibatch 2.

EDIT 2: I'm thinking this approach isn't incorrect, just perhaps a little odd. It would be similar, in a way, to a bidirectional LSTM (reading the input backward and forward to increase the exposure of the network to the end points - i.e. boost the signal). This also boosts the signal, but perhaps increases the bias by quite a bit. Thoughts?

Answer 1

Repeating samples will probably only slow down the learning process. As long as the mini batches are small this will probably not have a big effect. However, if batches and the number of repetitions become large gradient based learning can run into difficulties.

If you don't shuffle the batches, the repeated samples will prevent the model from taking in new information and may nudge it to "overfit" on the current samples. Since at every batch you only get one new sample (while all the others are repeated), the learning will either slow down or become stale and stuck at an unfavorable location.

Since your approach only has a two-dimensional input, this is probably not a major concern since the loss function will probably not be as complex as in other neural network applications.

Overall, in your case, you probably will not see a big impact on performance except for a longer learning time.

You do not need to be concerned with combining predictions for repeated samples, since you can make predictions using the final parameters after the learning is completed.