2
$\begingroup$

These days, I have seen many papers using intermediate supervision.

Single Network

When using a single neural network, multiple neurons output predictions, perhaps by processing data in different ways. Then, the loss function sums up the individual losses computed over each prediction.

For example, consider a part of the FlowNet architecture below:

A part of FlowNet

In this network, all the Convolution# layers (Convolution1, Convolution2, ...) output predictions at different stages of the network. Then, the loss function is computed by applying a mean-squared-error over all these predictions separately and adding them all.

Multiple Networks

When using multiple networks, like in Stacked Hourglass Networks:

enter image description here

Each individual network outputs a prediction, and the overall loss function is computed by computing a mean-squared error over each network's prediction and summing them all up.

My question is: what is the intuition behind doing this? I thought that this will force the first network to learn to predict the task well, while the remaining networks just perform identity transformations. Why is this not observed in practice?

Also, I have only seen this applied in CNNs, but I could be wrong.

$\endgroup$

1 Answer 1

2
$\begingroup$

Unfortunately there is no mathematical proof whether or not the intermediate supervision improve the performance of the network.

However stack hourglass article shows that according to ablation study: using more encoder – decoder units with intermediate supervision improve the performance – please have a look at the figure below.

In that case – the remaining network doesn't perform identity transformation (although there is a residual connection between the remaining networks).

My intuition is the every encoder – decoder serves as a PCA, that keep the most dominant feature maps. A series of encoder – decoder let the network reassess the feature maps until it get its most discriminate feature maps at the end. enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.