0
$\begingroup$

I am currently writing some stuff up on Long Short-term Memory cells (LSTM), and stumbled over a question which I had trouble answering on the fly. The LSTM takes as input $h_{t-1}$ (besides the cell state and the local input $x_t$), which is also the output of the previous time step $t-1$. I was looking for a definition of the term autoregressive, because in the context of ML, it is often used for models which make a prediction $y_t$ while knowing about all prior predictions $y_{t-n}$.

Wikipedia states that

The autoregressive model specifies that the output variable depends linearly on its own previous values and on a stochastic term (an imperfectly predictable term)

If I understand correctly, I would argue that the LSTM is a first-order autoregressive model, because it sees exactly one value emitted in the past. At the same time, I think it is not autoregressive, because (if I understand the definition correctly) it does not exclusively rely on previous time steps, but also receives local information in the form of $x_t$. Is anyone able to clarify whether the LSTM cell could be considered autoregressive or not?

$\endgroup$
0

1 Answer 1

1
$\begingroup$

It is autoregressive because its output at time $t - 1$, $h_{t - 1}$, is received as input for the computation at time $t$ and used to generate $h_t$. The fact that there are other inputs like $x_t$ does not make it not autoregressive.

In some configurations (e.g. language models), its predictions are even used as input $x$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.