1
$\begingroup$

Suppose I implement a supervised learning version of LSTM similar to this. Namely, I have these univariate time series data:

t Y
1 101
2 105
3 108
4 104
5 110
6 112
7 119
8 111
9 113
10 115

So Y at time 1 is 101, Y at time 2 is 105, etc.

The goal is to predict Y at time t+1 given Y at time t and t-1. That is, I can rearrange this data set to look like supervised learning:

Y_(t-1) Y_t Y_(t+1)
101 105 108
105 108 104
108 104 110
104 110 112
110 112 119
112 119 111
119 111 113
111 113 115

Notice here that each Y value from t=3 to t=10 (max) gets predicted only once.

Now say I partitioned my data set into overlapping minibatches of size 5. We'll consider only the first two minibatches for simplicity:

t Y
1 101
2 105
3 108
4 104
5 110

2 105
3 108
4 104
5 110
6 112

I do the same trick of converting this to a supervised learning problem for each minibatch:

Y_(t-1) Y_t Y_(t+1)
101 105 108
105 108 104
108 104 110

Y_(t-1) Y_t Y_(t+1)
105 108 104
108 104 110
104 110 112

Notice: now two of the predictions from the first minibatch are repeated in the second minibatch.

I know that if I had wanted to have only one prediction per Y value, then I should have converted the entire dataset to supervised first and then created minibatches.

Questions:

Is this alternative approach incorrect? If so, why? If not, should I average the predictions, or is there another accepted way of combining the predictions from several time steps like this?

EDIT: I should mention that the learned parameters of the model are different for each position, even though the data points are the same. For example, the parameters that would be used to predict Y_(t+1) = 104 in minibatch 1 are different from the parameters that would be used to predict Y_(t+1) = 104 in minibatch 2.

EDIT 2: I'm thinking this approach isn't incorrect, just perhaps a little odd. It would be similar, in a way, to a bidirectional LSTM (reading the input backward and forward to increase the exposure of the network to the end points - i.e. boost the signal). This also boosts the signal, but perhaps increases the bias by quite a bit. Thoughts?

$\endgroup$
  • $\begingroup$ if you shuffle the order of the mini batches it shouldn't make a difference (if I understand the setup correctly ;-) ) $\endgroup$ – oW_ Apr 23 '19 at 21:10
  • $\begingroup$ Hi @oW_, thanks for the thought. I'm not switching the order of the minibatches; I'm wondering if it's legit to change the series to a supervised learning format after creating minibatches (in which case I get multiple predictions of the same y). If that's okay, then how do I combine the predictions? $\endgroup$ – StatsSorceress Apr 23 '19 at 21:12
  • $\begingroup$ it's probably not a big deal if your mini batches are very small. if they become bigger and you have a lot of repetitions gradient based learning algorithms may "converge" early to an unfavorable point from which it needs to escape after the first mini batch has been phased out (and then it starts all over again). you probably will slow down any learning. I'm not sure why you would need to combine the predictions though? not sure if get what you mean by that $\endgroup$ – oW_ Apr 23 '19 at 21:19
  • $\begingroup$ Okay, that's basically what I suspected - not wrong, just inefficient! What I mean is that in the first minibatch, I have the sequence [105, 108] predicting the y value 104, and in the second minibatch I'm doing the same thing. Since the parameter values are different (a gradient update has happened between minibatches), there are going to be different predictions for the same y value. Can/should I combine these somehow? $\endgroup$ – StatsSorceress Apr 23 '19 at 21:25
  • $\begingroup$ you don't need to combine them. you can just use the parameters of the final model once training has completed. (however, you probably don't want to make predictions on your training data to avoid overfitting) $\endgroup$ – oW_ Apr 23 '19 at 21:29
2
$\begingroup$

Repeating samples will probably only slow down the learning process. As long as the mini batches are small this will probably not have a big effect. However, if batches and the number of repetitions become large gradient based learning can run into difficulties.

If you don't shuffle the batches, the repeated samples will prevent the model from taking in new information and may nudge it to "overfit" on the current samples. Since at every batch you only get one new sample (while all the others are repeated), the learning will either slow down or become stale and stuck at an unfavorable location.

Since your approach only has a two-dimensional input, this is probably not a major concern since the loss function will probably not be as complex as in other neural network applications.

Overall, in your case, you probably will not see a big impact on performance except for a longer learning time.

You do not need to be concerned with combining predictions for repeated samples, since you can make predictions using the final parameters after the learning is completed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.