3
$\begingroup$

I'm currently studying the former and have heard of the latter, and right now I'm thinking that they're the same. Are they?

$\endgroup$
7
$\begingroup$

Definitely they are different. Very deep nets have exploding/vanishing gradient problem. The authors of ResNet paper had seen that by stacking many layers of convolution and dense layers, the learning did not increase although they used ReLU activation and batch normalization. They used a concept named skip connection which helped the nets to learn whether the input to a typical layer should be preserved or it should be transformed by that layer. Using this concept allowed them to increase the number of layers without hesitating whether they would have vanishing/exploding gradients. The concept of residual nets was originally this. The paper uses this concept for spatial data but recently I've seen people debating using them in temporal cases too __time series data.

Recurrent nets are used in temporal domains. Tasks like sequence classification are examples of their usage. In this domain the net should know the information of previous seen data. Well known examples of these nets are LSTMS. Early recurrent nets had vanishing/exploding gradient problem too. but after years LSTMs get popular amongst deep-learning practitioners. They defined a concept named gates which could learn when to forget and when to keep the previous data.

$\endgroup$
4
$\begingroup$

No, they are not. Residual networks are deep networks with building blocks characterised by having skip connections across layers to facilitate backwards flow of gradients (or approximation of identity functions).

Recurrent networks are a different beast altogether, with cyclic connections, variable size inputs, ...

For residual nets, see the paper. For recurrent nets, see e.g. this wonderful post.

$\endgroup$
1
$\begingroup$

Both facilitate the input of information (not only from a neighbouring point, as is typical for all networks, but also) from an "earlier" point in the network.

What "earlier" means in each case is (one of the) main differences.

In case of ResNets this extra input happens by allowing info to "jump" from a point at a given depth to deeper point (layer) in the network (both corresponding to a single original input). If you consider the inputs are stacked vertically on the left of the network and the deeper network layers as proceeding from left to right horizontally, this would correspond to a "skipping" of some info from left to right at a given height. This helps information from the shallower depths of the network keep "alive" at further depths into the network. (Combats vanishing gradients etc)

In the case of a RNN this extra input takes the form of allowing a "jump" vertically at a certain horizontal position ie at a given depth (but varying input) This way the processing of a given input at some depth in the network can glean some information, not just from its input "ancestor" but also those of some "neigbouring" inputs. This can be especially useful if the inputs represent "sequences" which are related to one another like words in a sentence or notes in a melody - here the vertical is often considered a "temportal" (sequence) dimension

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.