Intuition / Importance of intermediate supervision in deep learning

Question

These days, I have seen many papers using intermediate supervision.

Single Network

When using a single neural network, multiple neurons output predictions, perhaps by processing data in different ways. Then, the loss function sums up the individual losses computed over each prediction.

For example, consider a part of the FlowNet architecture below:

In this network, all the Convolution# layers (Convolution1, Convolution2, ...) output predictions at different stages of the network. Then, the loss function is computed by applying a mean-squared-error over all these predictions separately and adding them all.

Multiple Networks

When using multiple networks, like in Stacked Hourglass Networks:

Each individual network outputs a prediction, and the overall loss function is computed by computing a mean-squared error over each network's prediction and summing them all up.

My question is: what is the intuition behind doing this? I thought that this will force the first network to learn to predict the task well, while the remaining networks just perform identity transformations. Why is this not observed in practice?

Also, I have only seen this applied in CNNs, but I could be wrong.

Answer 1

Unfortunately there is no mathematical proof whether or not the intermediate supervision improve the performance of the network.

However stack hourglass article shows that according to ablation study: using more encoder – decoder units with intermediate supervision improve the performance – please have a look at the figure below.

In that case – the remaining network doesn't perform identity transformation (although there is a residual connection between the remaining networks).

My intuition is the every encoder – decoder serves as a PCA, that keep the most dominant feature maps. A series of encoder – decoder let the network reassess the feature maps until it get its most discriminate feature maps at the end.