1
$\begingroup$

My question is regarding the transposed convolution operation (also commonly called deconvolution or upconvolution). In TensorFlow, for instance, I refer to this layer.

My question is, how / when do we add the bias (intercept) term when applying this layer?

When working with 'regular' convolution, we do this:

conv_output = tf.nn.conv2d(input, kernel, strides, padding='VALID')
conv_output = tf.nn.bias_add(conv_output, bias)

How do we do this when applying the deconvolution layer? My confusion arises because my advisor told me to visualise upconvolution as a pseudo-inverse convolutional layer (inverse in the sense that convolution downsamples the input, while transposed convolution upsamples it. I know they are not mathematically inverse.)

According to him:

Regular convolution: conv = x.w + b

Transposed convolution: x = (conv - b).W (where w and W are not the same).

Is the above equation even right? Something about it makes me feel uneasy.

In this scenario, since we are "going backwards", should we do something like this:

deconv_output = tf.nn.bias_add(input, -1 * bias)
deconv_output = tf.nn.conv2d_transpose(deconv_output, kernel, strides, padding='VALID')

Or should we add the bias after applying the transpose convolution, as we do in 'regular' convolution?

$\endgroup$
2
$\begingroup$

We are going backwards in the sense that we are upsampling and so doing the opposite to a standard conv layer, like you say, but we are more generally still moving forward in the neural network. For that reason I would add the bias after the convolution operations. This is standard practice: apply a matrix dot-product (a.k.a affine transformation) first, then add a bias before finally applying a non-linearity.

With a transpose convolution, we are not exactly reversing a forward (downsampling) convolution - such an operation would be referred to as the inverse convolution, or a deconvolution, within mathematics. We are performing a (transpose) convolution operation that returns the same input dimensions that produced the activation map in question, with no guarantee that the actual values are identical to the original input.

You can see from the animations of various convolutional operations here, that the transpose convolution is basically a normal convolution, but with added dilation/padding to obtain the desired output dimensions. The trick is to retain the mappings of localisation between the pixels.

In the paper from which those animations are taken, they explain how a transpose convolution is essentially the convolution steps performed in reverse:

..., although the kernel w defines a convolution whose forward and backward passes are computed by multiplying with $\textbf{C}$ and $\textbf{C}^T$ respectively, it also defines a transposed convolution whose forward and backward passes are computed by multiplying with $\textbf{C}^T$ and $(\textbf{C}^T)^T = \textbf{C}$ respectively.

One other source to back up my opinion: in the PyTorch implementation it seems the bias is added the output of the convolution's result.

$\endgroup$
3
  • $\begingroup$ Great answer. I was thinking along similar lines – that since we are still moving forward in the network, we must add the bias as we did before, especially to maintain universal approximation. $\endgroup$
    – James Bond
    Jun 25 '18 at 22:45
  • $\begingroup$ One question though (since you quote about transposed convolution in your answer): why exactly is this called transposed convolution? Is it because we use a transposed kernel? But what sense does it make to call a kernel transposed when it is to be learned? $\endgroup$
    – James Bond
    Jun 25 '18 at 22:48
  • 1
    $\begingroup$ It is the process which is transposed, not the weight matrix per se - we are going from activation maps to the input that produced them. Normally we get input and produce an activation map. $\endgroup$
    – n1k31t4
    Jun 25 '18 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.