Are there any differences between Recurrent Neural Networks and Residual Neural Networks?

Question

I'm currently studying the former and have heard of the latter, and right now I'm thinking that they're the same. Are they?

Answer 1

Definitely they are different. Very deep nets have exploding/vanishing gradient problem. The authors of ResNet paper had seen that by stacking many layers of convolution and dense layers, the learning did not increase although they used ReLU activation and batch normalization. They used a concept named skip connection which helped the nets to learn whether the input to a typical layer should be preserved or it should be transformed by that layer. Using this concept allowed them to increase the number of layers without hesitating whether they would have vanishing/exploding gradients. The concept of residual nets was originally this. The paper uses this concept for spatial data but recently I've seen people debating using them in temporal cases too __time series data.

Recurrent nets are used in temporal domains. Tasks like sequence classification are examples of their usage. In this domain the net should know the information of previous seen data. Well known examples of these nets are LSTMS. Early recurrent nets had vanishing/exploding gradient problem too. but after years LSTMs get popular amongst deep-learning practitioners. They defined a concept named gates which could learn when to forget and when to keep the previous data.

Answer 2

No, they are not. Residual networks are deep networks with building blocks characterised by having skip connections across layers to facilitate backwards flow of gradients (or approximation of identity functions).

Recurrent networks are a different beast altogether, with cyclic connections, variable size inputs, ...

For residual nets, see the paper. For recurrent nets, see e.g. this wonderful post.

Answer 3

Both facilitate the input of information (not only from a neighbouring point, as is typical for all networks, but also) from an "earlier" point in the network.

What "earlier" means in each case is (one of the) main differences.

In case of ResNets this extra input happens by allowing info to "jump" from a point at a given depth to deeper point (layer) in the network (both corresponding to a single original input). If you consider the inputs are stacked vertically on the left of the network and the deeper network layers as proceeding from left to right horizontally, this would correspond to a "skipping" of some info from left to right at a given height. This helps information from the shallower depths of the network keep "alive" at further depths into the network. (Combats vanishing gradients etc)

In the case of a RNN this extra input takes the form of allowing a "jump" vertically at a certain horizontal position ie at a given depth (but varying input) This way the processing of a given input at some depth in the network can glean some information, not just from its input "ancestor" but also those of some "neigbouring" inputs. This can be especially useful if the inputs represent "sequences" which are related to one another like words in a sentence or notes in a melody - here the vertical is often considered a "temportal" (sequence) dimension

Answer 4

I just wanted to add this.

• Recurrent neural networks (RNN) generally refer to the type of neural network architectures, where the input to a neuron can also include additional data input, along with the activation of the previous layer. E.g. for real-time handwriting or speech recognition.
• Residual neural networks (ResNet) refer to another type of neural network architecture, where the input to a neuron can include the activations of two (or more) of its predecessors. E.g. for non-realtime handwriting or speech recognition.

So, there is a difference but maybe these implementations can be considered equivalent, especially if we are performing non-real-time data analysis.