2
$\begingroup$

I'm studying fully convolutional neural networks for image segmentation, so far i've study and kind of understood the deconv network. Following this tutorial (Upsampling) i can't really understand how the "skip" method works.

I understood how to apply max unpooling and the transpose convolution for upsampling,but how does the "skip" method "relate" to those 2 methods?

Why do we use this "skip" method? Max unpooling and transpose convolution are not enough to upsample and give back a better resolution map?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

So as you already know, the convolutions extract some local information from the image, but for some decisions you need more global info. This is what pooling is supposed to do, it basically erases some locality information. In plain classification this is obviously necessary, because the question, if there is a cat in a picture, is a question about the entire image, not about one or a few pixels. This is why in the end of the classifier you get rid of all locality by adding fully-connected layers.

For semantic segmentation it is not quite as simple, because the output is supposed to still have locality information in it. You actually are just doing a pixel-wise classification and for each pixel the broad context matters. To decide, if a pixel is part of the cat in an image requires information about many pixels within a certain distance from the pixel in question. So this is why pooling layers are used for semantic segmentation as well. But pooling also reduces the image size, that's why upsampling and transpose convolutions are used to get back to the original size.

But there is a downside. As we said, pooling erases some information about the locality, in other words, it reduces the resolution. Once this information is gone, no combination of upsampling and transpose convolutions is going to get it back. Transpose convolutions can only learn to do this the least bad, kind of like a pseudo-inverse matrix. This limits the resolution/detail you can expect on your segmentation mask.

This problem is solved by skip connections by providing the relevant locality information where it is needed. Whenever you do upsampling, you find the last layer before the pooling, where the image still had the same size and simply add it pixel-wise to the upsampled image. This pixel-wise connection is implemented by using a 1x1-convolution in the link you posted. This way, the now upsampled feature map has both the locality detail from before pooling, but also the broad context info from after pooling. This allows for much better detail in the segmentation mask.

Improve this answer
$\endgroup$
3
  • $\begingroup$ So basically when we're performing the upsampling "section", we get back to the "symmetric" map in the downsampling part and then we sum the 2 maps pixel wise? $\endgroup$
    – Mattia Surricchio
    Commented Dec 9, 2019 at 10:33
  • 1
    $\begingroup$ Exactly! And the sum is weighted by the 1x1 convolution. I think in some architectures they go back to the "symmetric" map and concatenate it as an extra channel to the upsampled map, but the general idea is the same. $\endgroup$
    – matthiaw91
    Commented Dec 9, 2019 at 10:44
  • 1
    $\begingroup$ Yes, the Unet uses concatenation instead of sums $\endgroup$
    – Mattia Surricchio
    Commented Dec 9, 2019 at 10:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.