Adding bias in deconvolution (transposed convolution) layer

Question

My question is regarding the transposed convolution operation (also commonly called deconvolution or upconvolution). In TensorFlow, for instance, I refer to this layer.

My question is, how / when do we add the bias (intercept) term when applying this layer?

When working with 'regular' convolution, we do this:

conv_output = tf.nn.conv2d(input, kernel, strides, padding='VALID')
conv_output = tf.nn.bias_add(conv_output, bias)

How do we do this when applying the deconvolution layer? My confusion arises because my advisor told me to visualise upconvolution as a pseudo-inverse convolutional layer (inverse in the sense that convolution downsamples the input, while transposed convolution upsamples it. I know they are not mathematically inverse.)

According to him:

Regular convolution: conv = x.w + b

Transposed convolution: x = (conv - b).W (where w and W are not the same).

Is the above equation even right? Something about it makes me feel uneasy.

In this scenario, since we are "going backwards", should we do something like this:

deconv_output = tf.nn.bias_add(input, -1 * bias)
deconv_output = tf.nn.conv2d_transpose(deconv_output, kernel, strides, padding='VALID')

Or should we add the bias after applying the transpose convolution, as we do in 'regular' convolution?

Answer 1

We are going backwards in the sense that we are upsampling and so doing the opposite to a standard conv layer, like you say, but we are more generally still moving forward in the neural network. For that reason I would add the bias after the convolution operations. This is standard practice: apply a matrix dot-product (a.k.a affine transformation) first, then add a bias before finally applying a non-linearity.

With a transpose convolution, we are not exactly reversing a forward (downsampling) convolution - such an operation would be referred to as the inverse convolution, or a deconvolution, within mathematics. We are performing a (transpose) convolution operation that returns the same input dimensions that produced the activation map in question, with no guarantee that the actual values are identical to the original input.

You can see from the animations of various convolutional operations here, that the transpose convolution is basically a normal convolution, but with added dilation/padding to obtain the desired output dimensions. The trick is to retain the mappings of localisation between the pixels.

In the paper from which those animations are taken, they explain how a transpose convolution is essentially the convolution steps performed in reverse:

..., although the kernel w defines a convolution whose forward and backward passes are computed by multiplying with $\textbf{C}$ and $\textbf{C}^T$ respectively, it also defines a transposed convolution whose forward and backward passes are computed by multiplying with $\textbf{C}^T$ and $(\textbf{C}^T)^T = \textbf{C}$ respectively.

One other source to back up my opinion: in the PyTorch implementation it seems the bias is added the output of the convolution's result.