0
$\begingroup$

I have seen images of lst and rnn units online, where they "unravel" the unit. enter image description here

  1. Is this only one, singular, unit?
  2. If you have multiple units in a cell (layer), are both the cell state and hidden state carried through to the next unit? (or are they recycled in each unit)
  3. By ht and ht-1, I assume that all memories are stored in an array? (or is it 1 vector)
  4. I read in an article that the length of cell state and hidden state is equal to the amount of units in a cell (layer). If this is true, do each units output multiple predictions on the same thing or different things?

Image #2 (response to an answer) enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$
  1. The "unraveling" you are referring to is just to illustrate how the different time steps of the input are received and processed. It doesn't have anything to do with the number of units. The "number of units" actually refers to the dimensionality of the input vector and the hidden state.
  2. The output and hidden state are passed to the computation of the next time step.
  3. $h_t$ and $h_{t-1}$ and vectors that have been computed at different time steps. Depending on how you configure of the LSTM, you may get all $h_i$'s (e.g. to apply attention over them) or just the last one (e.g. to perform classification).
  4. As I mentioned in (1), the "number of units" actually refers to the dimensionality of the input vector and the hidden state so, what you read is true. The prediction at reach time step is a vector of real numbers.
Improve this answer
$\endgroup$
7
  • $\begingroup$ one second. i will read this later $\endgroup$
    – Walter
    Jul 26, 2022 at 15:56
  • $\begingroup$ each of these lstm squares (first image) loop back to only themselves, i assume. that would explain the arc above each blue circle in the second image. however, there is a line connecting each blue circle (verticle line). What would this line represent? Thank you. $\endgroup$
    – Walter
    Jul 26, 2022 at 16:10
  • 1
    $\begingroup$ The multiple rows represented on the bottom figure refer to the different components of the input, hidden state and output of the LSTM. The arrows between the different rows represent how each component is influenced by all the components of the previous column. In my opinion, this kind of representation is very confusing and leads to misconceptions of what neural networks are (i.e. a bunch of differentiable matrix operations). $\endgroup$
    – noe
    Jul 26, 2022 at 16:44
  • 1
    $\begingroup$ I think that the naming is confusing you. You'd better think of the LSTM as a black box that takes 3 inputs: $x$, $c$, and $h$ and generates 2 outputs $c'$ and $h'$. We apply the same LSTM multiple times over a sequence of $x$, that is, $x_1$, ...$x_n$ and we use the outputs $c'$ and $h'$ of the LSTM at a time step as input values for $c$ and $h$ of the next time step. $\endgroup$
    – noe
    Jul 27, 2022 at 7:10
  • 1
    $\begingroup$ When I say "apply multiple times", I mean apply along multiple time steps, each time step receiving one vector $x_t$ of the sequence $x_1,..., x_n$. As I mentioned before, the number of units is just the dimensionality of $x_i$, so the picture does not tell us how many units the LSTM has, just that it's being applied to at least 3 time steps: $t-1$, $t$ and $t+1$. $\endgroup$
    – noe
    Jul 27, 2022 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.