2
$\begingroup$

So I understand what "layers" are. If you have 5 layers in your model, your data basically gets transformed 5 times via 5 activation functions. The number of "neurons" within a layer dictate how many outputs a layer creates.

So what are "cells"? I never understood where "cells" come into play. Are they a collection of layers?

Per Wiki: https://en.wikipedia.org/wiki/Long_short-term_memory

If the orange are layers, then I would imagine each has a bunch of neurons. So a cell is a collection of layers and yellow stuff? I'm having trouble understanding where this "cell" fits into an overall NN architecture. I am used to the pictures with input layer -> hidden layer -> output layer. So where would the "cell" occur?

enter image description here

$\endgroup$
3
  • 2
    $\begingroup$ "Cell" is probably identical to "neuron," but it will be easier to answer accurately if you can point us to a use of "cell" in a paper/book/website etc. $\endgroup$ – bogovicj Aug 4 '20 at 19:12
  • $\begingroup$ Well the book "deeplearning" by goodfellow starts talking about "cells" the second we start talking about RNNs and LTSMs. I am not understanding that makes a cell distinct from layers and nuerons. Also wikipedia uses "cell" as well when it talks about LTSM en.wikipedia.org/wiki/Long_short-term_memory $\endgroup$ – user38283 Aug 4 '20 at 20:51
  • $\begingroup$ Thanks for clarifying. Turns out out cell != neuron for your purposes. $\endgroup$ – bogovicj Aug 5 '20 at 20:34
1
$\begingroup$

In short

A cell is in essence a forward neural network consisting of neurons and so $cell_{RNN}\neq neuron_{NN}$.

Rationale

In the context of recurrent neural networks, a layer consists of cells e.g. LSTM cells. Although they seem to position similarly in terms of architecture (figure), there is a fundamental difference between an RNN_cell and a NN_neuron.

  1. The number of RNN cells is generally equal to the sequence length (e.g. number of words in a sentence) or embedding output dimension (figure below). Conversely, the number of neurons in forward neural networks can generally be of any number.
  2. Each RNN_cell acts much line a NN itself. Specifically in LSTMs, each cell consists of hidden state $W_{hidden}$ and cell state $W_{cell}$ matrices. These are used to store weights and perform transformations in the various gates (forget, input, output) of the LSTM cell.

In essence, the transformations that take place in each cell are matrix multiplications of $W_{hidden}$ and $W_{cell}$ much like those in a forward NN. And so, the size of $W_{hidden}$ and $W_{cell}$ cannot be but the number of neurons of that NN which essentially determine the memory of the LSTM cells and model itself.

Figure 1

$\endgroup$
2
  • $\begingroup$ Thanks! I can't upvote you but otherwise I would. So a cell basically is a NN by itself. This would make sense as in my pic, there are layers each with their own activation functions. $\endgroup$ – user38283 Aug 6 '20 at 19:36
  • $\begingroup$ Happy to help. Yes in the context of sequence networks that is the case. In Keras LSTM you can define the size of the “NN” of each cell using the parameter units, a parameter used also in Dense layer that defines the size of a hidden layer dimension of a conventional forward NNs for example. $\endgroup$ – hH1sG0n3 Nov 13 '20 at 17:59
0
$\begingroup$

If you have 5 layers in your model, your data basically gets transformed 5 times via 5 activation functions.

That is not completely true. For dense neural networks (for example), your layers are made by neurons that each has its own activation function (so you'll have more than 5 activation functions). Each neuron transforms your data.

Now what is a cell ?

A cell is a mathematical function that receives one or more input, does the math, and gives one or more outputs.

First some examples:

  1. neuron: the result is the output of an activation function applied to the weighted sum of the previous layer
  2. convolution cell: passes the result of a convolution kernel multiplied to a relative range of the cells of the immediately previous layer to the next layer
  3. a lot of other cell examples in this post

Why are they called "cells" ? Because those mathematical function are used in artifical neural networks (ANN) context.

Why do they have a name ? Because they work well in (some) ANN, so people share their new mathematical function that helps an ANN to converge given a specific problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.