What is difference between feed forward neural network and LSTM?

Question

What is the difference between feed-forward neural network and LSTM? How do they differ in their architecture?

Answer 1

A feed-forward neural network looks like this:

input -> hidden layer 1 -> hidden layer 2 -> ... -> hidden layer k -> output.

Each layer may have a different number of neurons, but that's the architecture.

An LSTM (long-short term memory cell) is a special kind of node within a neural network. It can be put into a feedforward neural network, and it usually is. When that happens, the feedforward neural network is referred to as an LSTM (confusingly!).

So how does an LSTM work? Have a look at my answer here: Forget Layer in a Recurrent Neural Network (RNN) -

tl;dr: an LSTM cell has three gates, each of which is used to modulate its input in some way: an input gate, a forget gate, and an output gate. It has a "memory" and an output that get modified by the gates. That is, within a single LSTM cell:

(input & previous cell state) -> (input & forget gates) -> (update cell state)

(input & previous cell state & updated cell state) -> (output gate)

You can stack these cells in many different configurations. So there is no single "LSTM network" - rather a set of many possible architectures that can be built from these basic nodes.

Hope that gets you started!

Answer 2

As mentioned by Philipp, networks with feedback loops helps in modeling time in the data.

You would want to look this for an overview of various NN architectures: https://i.sstatic.net/LgmYv.png

Answer 3

Feed forward networks are networks where every node is connected with only nodes from the following layer. They don't have "circle" connections. Data can only travel from input to output without loops. Examples would be Simple Layer Perceptron or Multilayer Perceptrion. Convolutional Neural Networks also are purely feed forward networks

In opposition to that are recurrent neural networks. LSTM is one of those. These RNN can also be connected "sideways". Meaning that your data can travel not only forward into the next layer but also to other nodes in the same layer or backwards. From an architecture standpoint that means that although you have a network with maybe only one hidden layer, you get "deepness" by adding loops to nodes in that layer. Usually these represent "time" in the data.

This image shows what I mean by "deepness" through feedback loops: https://en.wikipedia.org/wiki/Recurrent_neural_network#/media/File:Recurrent_neural_network_unfold.svg

Although its technically one node in your layer architecture, it get's deeper and deeper the more loops you add

Answer 4

I want to discuss some high-level intuition behind LSTM networks. Here are some questions to help explore the why aspects:

• Why/when would we use an LSTM over a feed forward neural network (FFNN)?
• What are the pros and cons of LSTMs and how do they compare to FFNN?
• How are they different from traditional recurrent neural networks (RNNs)?

Feed Forward Neural Networks (FFNN)

Let us first consider a standard FFNN with architecture:

As you probably know, this FFNN takes three inputs, processes them using the hidden layer, and produces two outputs. We can expand this architecture to incorporate more hidden layers, but the basic concept still holds: inputs come in, they are processed in one direction, and they are outputted at the end.

This concept is very well explained by other articles, so I will not go into much more detail. The key takeaway is:

The primary condition that separates FFNN from recurrent architectures is that the inputs to a neuron must come from the layer before that neuron.

FFNN Wikipedia

Recurrent Neural Networks (RNN)

Recurrent neural networks are mathematically quite similar to FFNN models. Their main difference is that the restriction placed on FFNN is no longer applied:

Inputs to a neuron can come from any other layer.

You will often see this architecture is often "rolled" into a recurrent unit such as the following:

The "rolled" units you might see in architecture diagrams can therefore be deceptively small. When you unroll them, the network will often be quite deep!

RNN Wikipedia

Long-Short Term Memory (LSTM)

LSTMs are a special type of RNN that are designed to tackle the vanishing/exploding gradient problem.

When you train a traditional RNN, the network often suffers from vanishing/exploding gradients: unrolling a recurrent unit results in a very deep network! If you go through the backpropagation algorithm using traditional RNN architectures, the earlier layers will become less and less significant as we propagate through the network. This makes traditional RNNs prone to forgetting information, particularly data that appears many timesteps prior to the current time.

An example LSTM cell is illustrated as such:

This structure is similar to the traditional RNN unrolled unit, but the key difference with the LSTM are the gates: input gate, output gate, and forget gate. The function of these gates are well described by their names:

• input gate controls the data that enters the cell
• forget gate controls the extent to which data stays within the cell
• output gate controls the output of the cell via the activation function

This github IO post is a great introduction to the basics of LSTMs. It also does an amazing job of explaining the intuition behind the math of an LSTM.

LSTM Wikipedia

Properties and an Example Use Case of RNN

The feedback loops lend recurrent neural networks better to temporal challenges. Time is factored into their architecture!

Let us explore an example:

Perhaps you are using a network to predict the next word in a sentence. Say you are given the inputs:

Starving, Alice drives to the nearest store to buy [prediction]

A recurrent neural network might forget the first word "starving" whereas an LSTM would ideally propagate it. An LSTM would therefore use the context it heard previously in the sentence to guess "food" whereas an RNN might guess anything that is bought in a store, particularly given a long sentence or multiple sentences.

The gating mechanisms that allow this type of memory is explained well by @StatsSorceress here: Forget Layer in a Recurrent Neural Network (RNN) -

RNNs are designed to handle sequences. This can be used to analyze video (sequences of images), writing/speech (sequences of words), etc.

LSTMs are designed to let important information persist over time. RNNs will often "forget" over time.

FFNNs are memoryless systems; after processing some input, they forget everything about that input. Say, for example, we train an FFNN that takes 5 words as inputs and predicts the next output. This model would then receive the input from the above example:

the nearest store to buy [prediction].

This is clearly losing context, and we would get a poor result.

Now, you might ask, what if we made a FFNN that took many inputs so that it included the word "starving" in its inputs? In other words, could we not increase the number of inputs to an FFNN to sufficiently represent prior data?

The answer is yes, but this method is both inefficient and often impractical. Say, for example, we need to remember context an entire paragraph back. We would need a very large FFNN! Moreover, how do we know how many words back we want as context? Would 20 be sufficient? 30? 100? The LSTM architecture eliminates these problems entirely by letting the network manage a transient memory. This problem is exacerbated by video processing challenges because each frame will require significantly more inputs than NLP tasks.

Note, however, that LSTM units are inherently more computationally complex than units of a FFNN, so training them is often more difficult and takes more time. Despite these disadvantages, their success in temporal data challenges clearly warrants their use over FFNN.