These days, I have seen many papers using intermediate supervision.
Single Network
When using a single neural network, multiple neurons output predictions, perhaps by processing data in different ways. Then, the loss function sums up the individual losses computed over each prediction.
For example, consider a part of the FlowNet architecture below:
In this network, all the Convolution#
layers (Convolution1
, Convolution2
, ...) output predictions at different stages of the network. Then, the loss function is computed by applying a mean-squared-error over all these predictions separately and adding them all.
Multiple Networks
When using multiple networks, like in Stacked Hourglass Networks:
Each individual network outputs a prediction, and the overall loss function is computed by computing a mean-squared error over each network's prediction and summing them all up.
My question is: what is the intuition behind doing this? I thought that this will force the first network to learn to predict the task well, while the remaining networks just perform identity transformations. Why is this not observed in practice?
Also, I have only seen this applied in CNNs, but I could be wrong.