What is the difference between Dilated Convolution and Deconvolution?

Question

These two convolution operations are very common in deep learning right now.

I read about dilated convolutional layer in this paper : WAVENET: A GENERATIVE MODEL FOR RAW AUDIO

and De-convolution is in this paper : Fully Convolutional Networks for Semantic Segmentation

Both seem to up-sample the image but what is the difference?

Answer 1

In sort of mechanistic/pictorial/image-based terms:

Dilation: ### SEE COMMENTS, WORKING ON CORRECTING THIS SECTION

Dilation is largely the same as run-of-the-mill convolution (frankly so is deconvolution), except that it introduces gaps into it's kernels, i.e. whereas a standard kernel would typically slide over contiguous sections of the input, it's dilated counterpart may, for instance, "encircle" a larger section of the image --while still only have as many weights/inputs as the standard form.

(Note well, whereas dilation injects zeros into it's kernel in order to more quickly decrease the facial dimensions/resolution of it's output, transpose convolution injects zeroes into it's input in order to increase the resolution of it's output.)

To make this more concrete, let's take a very simple example:
Say you have a 9x9 image, x with no padding. If you take a standard 3x3 kernel, with stride 2, the first subset of concern from the input will be x[0:2, 0:2], and all nine points within these bounds will be considered by the kernel. You would then sweep over x[0:2, 2:4] and so on.

Clearly, the output will have smaller facial dimensions, specifically 4x4. Thus, the next layer's neurons have receptive fields in the exact size of these kernels passes. But if you need or desire neurons with more global spatial knowledge (e.g if an important feature is only definable in regions larger than this) then you will need to convolve this layer a second time to create a third layer in which the effective receptive field is some union of the previous layers r.f..

But if you don't want to add more layers and/or you feel that the information being passed on is overly redundant (i.e. your 3x3 receptive fields in the second layer only actually carry "2x2" amount of distinct information), you can use a dilated filter. Let's be extreme about this for clarity and say we'll use a 9x9 3-dialated filter. Now, our filter will "encircle" the entire input, so we won't have to slide it at all. We will still however, only be taking 3x3=9 data points from the input, x, typically:

x[0,0] U x[0,4] U x[0,8] U x[4,0] U x[4,4] U x[4,8] U x[8,0] U x[8,4] U x[8,8]

Now, the neuron in our next layer (we'll only have one) will have data "representing" a much larger portion of our image, and again, if the image's data is highly redundant for adjacent data, we may well have preserved the same information and learned an equivalent transformation, but with fewer layers and fewer parameters. I think within the confines of this description it's clear that while definable as resampling, we are here downsampling for each kernel.

Fractionally-strided or transpose or "deconvolution":

This sort is very much still convolution at heart. The difference is, again, that we will be moving from smaller input volume to a larger output volume. OP posed no questions about what upsampling is, so I'll save a bit of breadth, this time 'round and go straight to the relevant example.

In our 9x9 case from before, say we want to now upsample to 11x11. In this case, we have two common options: we can take a 3x3 kernel and with stride 1 and sweep it over our 3x3 input with 2-padding so that our first pass will be over the region [left-pad-2:1, above-pad-2: 1] then [left-pad-1:2, above-pad-2: 1] and so on and so forth.

Alternatively, we can additionally insert padding in between the input data, and sweep the kernel over it without as much padding. Clearly we will sometimes be concerning ourselves with the exact same input points more than once for a single kernel; this is where the term "fractionally-strided" seems more well-reasoned. I think the following animation (borrowed from here and based (I believe) off of this work will help to clear things up despite being of different dimensions. The input is blue, the white injected zeros and padding, and the output green:

Of course, we're concerning ourselves with all of the input data as opposed to dilation which may or may not ignore some regions entirely. And since we're clearly winding up with more data than we began, "upsampling".

I encourage you to read the excellent document I linked to for a more sound, abstract definition and explanation of transpose convolution, as well as for learning why the examples shared are illustrative but largely inappropriate forms for actually computing the represented transformation.

Answer 2

Though both seem to be doing the same thing, which is up-sampling a layer, there's a clear margin between them.

First we talk about Dilated Convolution

I found this nice blog on above topic. So as I understood, this is more like exploring the input data points in a wide manner. Or increasing the receptive field of the convolution operation.

Here is a dilated convolution diagram from the paper.

This is more of normal convolution, but help to capture more and more global context from input pixels without increasing the size of the parameters. This can also help to increase the spacial size of the output. But the main thing here is this increases the receptive field size exponentially with the number of layers. This is very common in the signal processing field.

This blog really explains what is new in the dilated convolution and how this is compared to normal convolution.

Blog: Dilated Convolutions and Kronecker Factored Convolutions

Now I will explain what is Deconvolution

This is called transposed convolution. This is equal to the function we used for convolution in the back-propagation.

Simply in backprop we distribute gradients from one neuron in the output feature map to all the elements in the receptive fields, then we also sum up gradients for where they coincided with same receptive elements

Here's a good resource with pictures.

So the basic idea is deconvolution works in the output space. Not input pixels. It will try to create more broader spacial dimensions in the output map. This is used in Fully Convolutional Neural Nets for Semantic Segmentation.

So more of Deconvolution is a learnable up-sampling layer.

It's tries to learn how to up-sample while combining with the final loss

This is the best explanation I found for deconvolution. Lecture 13 in cs231, from 21.21 onward.

Answer 3

The crux of this question is to know more about dilated convolutions(I guess). So, I am assuming everyone is familiar with the normal convolution operation. I'll mention it very briefly down below.

Dilated/Atrous Convolutions

• Useful when we want to generate output by looking at a larger field of view.
• Extremely popular in the field of image segmentation.
• Dilation is a process by which we increase the receptive field of the convolution(receptive field ~ Field of view). Original convolving filter :

For dilation : just move each red block around the center by k-1 units away(if the dilation rate = k ) and fill the empty slots by 0.

• Dilation rate may be varied accordingly to visualize larger areas.
• Dilated convolutions support the exponentiation of receptive fields without loss of expansion/coverage.
• Moreover, no additional computation resources are burned. There is no resolution loss unlike pooling and strided convolutions.