0
$\begingroup$

I was following Tensorflow's own time series/LSTM tutorial, and there's something I don't quite understand about the whole process around Backpropagation Through Time (BPTT).

The resources I've found explain how prediction for a time in a future with an offset work (the "offset" in the tutorial, and the t+3rd time instance in the Wikipedia article)

What I don't understand is how does this generalize to where the input width is more than 1?

So in the Wikipedia article we see the case for inputWidth=1, outputWidth=1 and offset=3.

I think I understand if outputWidth>1 then you'd just start using not only the last, but also previous cell predictions as output. I also understand if offset>1 the you just unroll the network into offset number of steps (as shown on Wikipedia).

But what if inputWidth>1? Let's say inputWidth=k (k>1)!

In this case, do you unroll the same RNN multiple times in parallel, and somehow average the weights? Do you unroll the same RNN k times sequentially? What does that look like?

Improve this question
$\endgroup$
2
  • $\begingroup$ What do you mean with "width"? Do you mean "length"? $\endgroup$
    – noe
    Commented Dec 2, 2020 at 20:22
  • $\begingroup$ @ncasas yeah, basically with how many timesteps you pass as input. Check the linked resources, should be self-explanatory. $\endgroup$
    – lte__
    Commented Dec 3, 2020 at 21:36

1 Answer 1

Reset to default
0
+50
$\begingroup$

You unroll the same RNN multiple times, but not in parallel. The RNN needs both an input and the previous step, therefore, you unroll the same RNN for the first time step, then for the second, and so on.

For each step it is always the same RNN, so the weights are the same each time:

enter image description here

Improve this answer
$\endgroup$
3
  • $\begingroup$ I think what this image shows is just showing inputWidth=1, outputWidth=5 and offset=5. You're taking x_t1, x_t2, x_t3, x_t4 and x_t5 (in red), putting it through the RNN (light blue) with shared weights and predict y_t1, y_t2, y_t3, y_t4 and y_t5 (in dark blue). I'm interested in how it works when you can use x1, x2 and x3, (inputWidth=3)(!!) with an offset of let's say 2and maybe outputWidth=1. So the input is x_t1 & x_t2, (with y_t3 as the label) x_t2& x_t3 (with y_t4 as the label), and x_t3 &x_t4 (with y_t5 as the label). $\endgroup$
    – lte__
    Commented Dec 13, 2020 at 11:42
  • $\begingroup$ You are giving too much importance to inputWidth, outputWidth and offset, but the RNN always works the same regardless of them. These parameters are specific to that tutorial and are only meant to define input and output data in that very case. Your inputWidth is just the length of the input, and therefore it is also for how many time steps the output of the RNN is ignored. outputWidth is just for how many timesteps you zero-out the input once the input has been consumed. $\endgroup$
    – noe
    Commented Dec 13, 2020 at 12:18
  • $\begingroup$ Note that an LSTM can be used in many ways, not only the way you suggest. It is possible to have the LSTM ingest inputs and, at each timestep $t$ (starting from $t=0$) make the output be the prediction for $t+1$. It is possible to have the RNN ingest many input timesteps and after that, have it predict a single value, or multiple values (zero'ing out the input for those timesteps). $\endgroup$
    – noe
    Commented Dec 13, 2020 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.