# Philosophical question on redundancy

Suppose I implement a supervised learning version of LSTM similar to this. Namely, I have these univariate time series data:

t Y
1 101
2 105
3 108
4 104
5 110
6 112
7 119
8 111
9 113
10 115


So Y at time 1 is 101, Y at time 2 is 105, etc.

The goal is to predict Y at time t+1 given Y at time t and t-1. That is, I can rearrange this data set to look like supervised learning:

Y_(t-1) Y_t Y_(t+1)
101 105 108
105 108 104
108 104 110
104 110 112
110 112 119
112 119 111
119 111 113
111 113 115


Notice here that each Y value from t=3 to t=10 (max) gets predicted only once.

Now say I partitioned my data set into overlapping minibatches of size 5. We'll consider only the first two minibatches for simplicity:

t Y
1 101
2 105
3 108
4 104
5 110

2 105
3 108
4 104
5 110
6 112


I do the same trick of converting this to a supervised learning problem for each minibatch:

Y_(t-1) Y_t Y_(t+1)
101 105 108
105 108 104
108 104 110

Y_(t-1) Y_t Y_(t+1)
105 108 104
108 104 110
104 110 112


Notice: now two of the predictions from the first minibatch are repeated in the second minibatch.

I know that if I had wanted to have only one prediction per Y value, then I should have converted the entire dataset to supervised first and then created minibatches.

Questions:

Is this alternative approach incorrect? If so, why? If not, should I average the predictions, or is there another accepted way of combining the predictions from several time steps like this?

EDIT: I should mention that the learned parameters of the model are different for each position, even though the data points are the same. For example, the parameters that would be used to predict Y_(t+1) = 104 in minibatch 1 are different from the parameters that would be used to predict Y_(t+1) = 104 in minibatch 2.

EDIT 2: I'm thinking this approach isn't incorrect, just perhaps a little odd. It would be similar, in a way, to a bidirectional LSTM (reading the input backward and forward to increase the exposure of the network to the end points - i.e. boost the signal). This also boosts the signal, but perhaps increases the bias by quite a bit. Thoughts?

• if you shuffle the order of the mini batches it shouldn't make a difference (if I understand the setup correctly ;-) ) – oW_ Apr 23 '19 at 21:10
• Hi @oW_, thanks for the thought. I'm not switching the order of the minibatches; I'm wondering if it's legit to change the series to a supervised learning format after creating minibatches (in which case I get multiple predictions of the same y). If that's okay, then how do I combine the predictions? – StatsSorceress Apr 23 '19 at 21:12
• it's probably not a big deal if your mini batches are very small. if they become bigger and you have a lot of repetitions gradient based learning algorithms may "converge" early to an unfavorable point from which it needs to escape after the first mini batch has been phased out (and then it starts all over again). you probably will slow down any learning. I'm not sure why you would need to combine the predictions though? not sure if get what you mean by that – oW_ Apr 23 '19 at 21:19
• Okay, that's basically what I suspected - not wrong, just inefficient! What I mean is that in the first minibatch, I have the sequence [105, 108] predicting the y value 104, and in the second minibatch I'm doing the same thing. Since the parameter values are different (a gradient update has happened between minibatches), there are going to be different predictions for the same y value. Can/should I combine these somehow? – StatsSorceress Apr 23 '19 at 21:25
• you don't need to combine them. you can just use the parameters of the final model once training has completed. (however, you probably don't want to make predictions on your training data to avoid overfitting) – oW_ Apr 23 '19 at 21:29