0
$\begingroup$

Consider an LSTM model with 100 timesteps, each of which with input and target data. Let f(99) be the function mapping the input data of the 99th timestep and hidden state of the 98th timestep to the output data of the 99th timestep.

The mapping f(99) of another LSTM model will be different if the other LSTM model is fitted only to the data in the first 99 timesteps. It follows that future data -- that is, data from the 100th timestep -- was used to generate the 99th timestep's prediction in the first LSTM model.

What are the implications of this for forecasting methodology?

Improve this question
$\endgroup$
1
$\begingroup$

I think you have assumed that future timesteps also are used for learning by a simple LSTM. But, that actually is Bidirectional LSTM which learns from the past as well as future timesteps. Simple Recurrent Neural Networks don't have the access to future input information. You might want to refer to a book on recurrent neural networks. You can also look this question on cross validated for training an LSTM.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for this. But isn't it the case that the loss of a RNN is the sum of the losses associated with all of the timesteps? So then the loss associated with a future timestep (t+1) has an impact on the output associated with timestep t. I was wondering if it was problematic that there is an implicit dependency on future timesteps' loss functions but was at the same time aware that its output is only explicitly dependent on the hidden state of past timesteps. $\endgroup$ – Solver Sep 15 '18 at 18:18
  • $\begingroup$ I believe you are talking about many-to-many classification problem, e.g. segment classification. You are correct that it becomes problematic for simple RNNs. For these kind of problems BRNNs have been outperforming simple RNNs. $\endgroup$ – naive Sep 15 '18 at 19:35
  • $\begingroup$ You're right, but in my specific case, many-to-many regression. But I was specifically thinking about this in the context of forecasting $\endgroup$ – Solver Sep 15 '18 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.