In classical neural nets, we have that each layer is connected only with the following layer. What if we relaxed this constraint and allowed it to be connected to any or all subsequent layers? Has this architecture been explored? It seems to be backprop would still work on this.
At the very least, this type of network could be emulated by artificially creating identity neurons at each layer that bring in the value of every earlier node, where the incoming weights to those layers are fixed at 1.