In classical neural nets, we have that each layer is connected only with the following layer. What if we relaxed this constraint and allowed it to be connected to any or all subsequent layers? Has this architecture been explored? It seems to be backprop would still work on this.

At the very least, this type of network could be emulated by artificially creating identity neurons at each layer that bring in the value of every earlier node, where the incoming weights to those layers are fixed at 1.


What you describe has been explored in Deep Residual Neural Networks.

A residual block will combine two or more blocks from a standard architecture like a CNN with a skip connection that adds the input to the first block to the output of the last block.

enter image description here

The intuition is that deep networks have a harder and harder time learning the identity function between layers, which has been proven to be useful especially in image recognition tasks. Residual connections also mitigate the problem of vanishing gradients.

Residual connections help solve the "degradation" problem, where deeper architectures lead to reduced accuracy. For example GoogLeNet won ILSVRC in 2014 with a 22-layer CNN, but in 2015 Microsoft ResNet won with a 152-layer Res Net.

  • 1
    $\begingroup$ Thanks for the link. This intuitively makes sense to me, as now each layer has access to all input data, not just the last layers encoding. Of course, this has to be at least as good as having the subset of data. $\endgroup$
    – dashnick
    Feb 5 '18 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.