10
$\begingroup$

I'm trying to learn how LSTM networks work, and even if I get the basics, the details of the internal structure is not clear for me.

On this blog link, I found this scheme of a LSTM architecture

LSTM

Where apparently, every circle should correspond to an individual LSTM unit like this

LSTM

  1. Is this correct?

  2. Is each unit in the cell independent from the others? Or do they share information?

  3. Imagine I have the following configuration: Number of samples = 1000 Number of time-steps = 10 Number of features = 5

In this case, each unit in a cell will take as input a vector of size 5 right?

But what will be the size of the output for one unit? 1?

Thanks

$\endgroup$
1

3 Answers 3

2
$\begingroup$
  1. Is this correct?

Yes the diagrams both look correct to me. The key thing to understanding both diagrams is that the inputs and outputs of an LSTM cell are vectors.

The circles in the first diagram do represent the concept that the layer contains multiple individual artificial neurons, and that might make you assume that the second diagram is a picture of one of those neurons. Arguably there are multiple "neurons" or sub-layers with different roles inside a cell, because there are multiple places in which calculations of the form $f(W\mathbf{x} +b)$ occur, performing slightly different roles. I think the term "cell" is used to refer to this architecture of neurons. As a short-hand when I say "neuron", I tend to think of the output stage of the hidden layer.

However, in the second diagram, all of the operations shown work with vectors. Most importantly the left-to-right arrows in the second diagram represent vectors of hidden state from timestep to timestep from the whole layer. So each neuron in the cell is recurrently connected to every other neuron in that cell - twice as it happens in an LSTM, because LSTMs have both an internal cell state and a layer output.

  1. Is each unit in the cell independent from the others? Or do they share information?

To match your description of the diagrams, let's define a "unit" as a collection of one of each type of neuron/gate used to make up the cell, that in theory could be wired together to make a working LSTM cell layer with a single scalar cell state and output value.

These units are independent in that each has its own weight parameters. There are no shared parameters for the connections between input and the units, or for the recursive connections that forward state from one time step to the next. In that sense the units do not share information.

However, the connections do mean that on each time step, input data and hidden state plus output from last output from all other units in the cell are combined are used in calculations. Any cell unit can base its new internal state plus its output on the values of all other outputs and internal states from other units in the cell. In this sense, the units do share information. I guess from your question that it is probably this second issue that you are concerned about, as the second diagram makes you think of a wiring diagram for a single neuron, but as explained above that is not the case.

  1. Imagine I have the following configuration: Number of samples = 1000 Number of time-steps = 10 Number of features = 5 In this case, each unit in a cell will take as input a vector of size 5 right?

Almost. Each neuron inside the cell will take an input of 5 from $\mathbf{x}$, plus an input of the hidden layer output, $\mathbf{h}$. So if in your case the LSTM cell size was 10, then each neuron would take a combined vector of 15. In addition, a second cell state vector is maintained, not labelled in your diagram. That is not used directly as an input to any neurons (i.e. components of the form $f(W\mathbf{x}+b)$), but does interact with the other values and can get changed itself, through the various gates. In your second diagram, it is the uppermost arrow going left to right.

But what will be the size of the output for one unit? 1?

The cell as a whole will have the output of whatever size you have made the layer. That is what the diagram number 2 tries to show. However, using our working definition of "unit", the output of each unit will be two scalar values - the hidden layer output, and the cell state - which will be part of their respective vectors shown in the diagrams.

$\endgroup$
1
  • $\begingroup$ thank you for the answer, but it seems that first OP's question should be answered "No". $unit$ seems to be analog of feed-forward $node$, or $neuron$. So second picture looks like a "cell" which, as you correctly admit, corresponds to $layer$ concept (there is the same discussion in comments of OP's link post). $\endgroup$
    – garej
    Jun 21, 2020 at 19:47
1
$\begingroup$

I think your image is mislabeled. I think each blue box is an LSTM layer, composed of multiple cells/units, each of which accepts a vector input x_t.

With that, the answers to your questions are:

1) Yes

2) Yes, they are independent (at a single time step - they share information with each other between time steps)

3) Yes, each unit cell will take an input of size 5. I think the output size is always 1, similar to neural network nodes (like sigmoidal units) that combine and then activate.

$\endgroup$
2
  • $\begingroup$ I confirm that the output size is always one, because each LSTM unit esentially works similar to ordinary neuron. It's up to the following layer to distribute this single value to its multiple neurons $\endgroup$
    – Kari
    Mar 7, 2018 at 21:31
  • 1
    $\begingroup$ An "LSTM layer" is also called an "LSTM cell" AFAIK. When you add an LSTM cell in most frameworks you get to specify the layer size. So the labelling does appear to follow that convention. $\endgroup$ Mar 8, 2018 at 8:01
1
$\begingroup$

Actually the shapes are for simplification. If you want to know the correct behavior you have to take a look at the formulas of each LSTM cell. To answer your first question, There may be different answers. What the pictures are depicting belongs to tasks which are many to many, and for each input you need exactly one output. There are different tasks for sequences that can be defined:

  • One to one
  • One to many
  • Many to one
  • Many to many

You can take a look at here.

For illustrating the formulas of each LSTM cell I've given the following picture from professor Andrew Ng course about deep learning:

enter image description here

As you can see, each node in the LSTM cell can be connected indirectly to the output of the adjacent cells of the previous time-step. It's indirect because there are gates between them. Also consider that LSTM cell shares the weights for all the inputs of different time steps. Consequently each neuron in LSTM cell is dependent to the input of the current time-step and the output of the adjacent nodes of the previous time-steps.

About the third question, the input size will be equal to the number of features of the input for each time-step. The number of the outputs depends on your task as I referred to at first. Take a look at the first link.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.