It is my impression that a single layer LSTM architecture consists of $t$ LSTM cells that are identical duplicates, where $t$ is the number of time steps. Then there are gates within the LSTM cell. I have struggled to find a rigorous explanation of what each “gate” actually consists of. Is each gate simply a feed forward neural network who’s output is either squished through sigmoid or tanh depending on which gate it is?



1 Answer 1


Is each gate simply a feed forward neural network who’s output is either squished through sigmoid or tanh depending on which gate it is?

Close. Each gate is an activation function (typically sigmoid for actual gates, but tanh is used for other functions within the cell) over a weighted sum of all inputs to the cell's layer. Although spome reading of the diagrams may imply LSTM cells are processing a few single inputs in isolation, recombined later, in fact the inputs to each cell are the whole previous layer and the whole previous timestep cell states for the whole layer. This is similar to a view of a single neuron in a fully-connected feed-forward network.

Each gate in each cell has its own set of weights - one weight for each input from previous layer plus one weight for each cell state in the same layer, plus a single bias. When you combine these cells into a layer of multiple cells (e.g. when you choose to have an LSTM layer with 64 cells), this results in a separate matrix plus bias for each gate. In descriptions of LSTMs, these different matrices are often named for the type of gate they parametrise, so e.g. there will be a matrix plus bias for the layer's "forget" gate, which might be noted as $\mathbf{W}_f$.

If you have $N_i$ inputs, and $N_c$ LSTM cells in a single LSTM layer, then $\mathbf{W}_f$ will be a $N_c \times (N_i + N_c)$ matrix of weights, and so will all the other parameter matrices describing other gates and value calculations for the combined cells.

In practice the calculations don't need to be handled per cell, but calculations over a layer of cells can be vectorised, and it is more like having a few parallel fully connected layers that combine in various ways to generate output plus next cell states.

There would be nothing stopping you extending this architecture and making any single gate or value calculation deeper by giving its own hidden layers. I suspect this has been tried by researchers, but cannot find any references. But without this customisation, the gates in standard LSTM are more like logistic regression over concatenated input and cell state.

  • $\begingroup$ Thanks for the reply! Would you mind please elaborating on a “cell layer”. What actual parameters are being updated? I thought that the LSTM “forget gate” would learn “when to forget” for example. Are the LSTM cells not simply arranged linearly as most diagrams depict? $\endgroup$ Oct 31, 2019 at 13:16
  • $\begingroup$ @HarryStuart: The diagrams are accurate, but I wanted to note that each cell takes as input all previous layer input and all previous timestep cell states. That is not always clear. By "cell layer" I am probably noty using the correct terminology, but I mean that if you choose to have e.g. 64 cells that they are typically arranged together, taking the same inputs and feeding into the same output (which might be a normal feed-forward layer, or another group of LSTM cells). I will add a bit about the parameters. $\endgroup$ Oct 31, 2019 at 18:34
  • $\begingroup$ Ahhhhh. I think I understand now. I was struggling with seeing where an LSTM network got its flexibility from, but evidently it’s from the fact that the gates have respective weight matrices. I might write up an LSTM in tensorflow and print out the weight matrices etc. just to consolidate. Thanks for the help! $\endgroup$ Nov 2, 2019 at 2:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.