Why do we need to concatenate in a U-Net?

Question

You might be familiar with the U-Net, a machine learning network deceived for image segmentation. It's basically an encoder/decoder network with some direct links between encoder and decoder segments:

I would like to better understand the reason behind those gray lines!

What is the idea behind copying one part (cropped) of the outputs of the convolutional segments (before max-pooling) and concatenating it to the inputs to the individual up-convolutional layer?

Answer 1

These types of connections between layers are called skip connections, searching for this will give you a way to find more in-depth information. Broadly speaking, there are two advantages to using skip connections in this case:

• They allow gradients to more freely flow through the model, mitigating the issue of vanishing gradients
• They allow features from the encoder side of the network to the decoder side of the network, adding extra information that might be lost because of the downsampling on the encoder side of the network.

Answer 2

The information at the first layers of the network can get "muddy" as it passes through to the final layers of the network and skip connections are a great way to remind the network what it saw earlier.

One really interesting way to see this effect is to build an image autoencoder with and without these concatenations. Without them (a regular encode/decode autoencoder), your reconstructed input image is going to look awful at first and take a lot of training to start looking good. All the information to make the reconstruction has to fit inside the bottleneck layer in the middle.

Now add skip connections and the network can almost immediately reconstruct your image with high accuracy. All the information for the reconstruction no longer has to fit in the middle layer. Information from early on can be shared directly with the output layer.