13
$\begingroup$

I have the following CNN:

network layour

  1. I start with an input image of size 5x5
  2. Then I apply convolution using 2x2 kernel and stride = 1, that produces feature map of size 4x4.
  3. Then I apply 2x2 max-pooling with stride = 2, that reduces feature map to size 2x2.
  4. Then I apply logistic sigmoid.
  5. Then one fully connected layer with 2 neurons.
  6. And an output layer.

For the sake of simplicity, let's assume I have already completed the forward pass and computed δH1=0.25 and δH2=-0.15

So after the complete forward pass and partially completed backward pass my network looks like this:

network after forward pass

Then I compute deltas for non-linear layer (logistic sigmoid):

$$ \begin{align} &\delta_{11}=(0.25 * 0.61 + -0.15 * 0.02) * 0.58 * (1 - 0.58) = 0.0364182\\ &\delta_{12}=(0.25 * 0.82 + -0.15 * -0.50) * 0.57 * (1 - 0.57) = 0.068628\\ &\delta_{21}=(0.25 * 0.96 + -0.15 * 0.23) * 0.65 * (1 - 0.65) = 0.04675125\\ &\delta_{22}=(0.25 * -1.00 + -0.15 * 0.17) * 0.55 * (1 - 0.55) = -0.06818625\\ \end{align} $$

Then, I propagate deltas to 4x4 layer and set all the values which were filtered out by max-pooling to 0 and gradient map look like this:

enter image description here

How do I update kernel weights from there? And if my network had another convolutional layer prior to 5x5, what values should I use to update it kernel weights? And overall, is my calculation correct?

$\endgroup$
  • $\begingroup$ Please clarify what's confusing you. You already know how to do the derivative of the maximum (everything is zero except where the value is maximum). So, let's forget max-pooling. Is your problem in the convolution? Each convolution patch will have their own derivatives, it's a slow computational process. $\endgroup$ – Ricardo Cruz Feb 9 '18 at 0:05
  • $\begingroup$ The best source is the deep learning book -- admittedly not an easy read :). The first convolution is the same thing as dividing the image in patches and then applying a normal neural network, where each pixel is connected to the number of "filters" you have using a weight. $\endgroup$ – Ricardo Cruz Feb 10 '18 at 16:38
  • 1
    $\begingroup$ Is your question in essence how are kernel weights adjusted by using backpropagation? $\endgroup$ – JahKnows Feb 13 '18 at 2:01
  • $\begingroup$ @JahKnows ..and how gradients are calculated for convolutional layer, given the example in question. $\endgroup$ – koryakinp Feb 13 '18 at 2:35
  • $\begingroup$ Is there an activation function associated with your convolutional layers? $\endgroup$ – JahKnows Feb 13 '18 at 3:04
6
+100
$\begingroup$

A convolution employs a weight sharing principle which will complicate the mathematics significantly but let's try to get through the weeds. I am drawing most of my explanation from this source.


Forward pass

As you observed the forward pass of the convolutional layer can be expressed as

$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$

where in our case $k_1$ and $k_2$ is the size of the kernel, in our case $k_1=k_2=2$. So this says for an output $x_{0,0} = 0.25$ like you found. $m$ and $n$ iterate across the dimensions of the kernel.

Backpropagation

Assuming you are using the mean squared error (MSE) defined as

$E = \frac{1}{2}\sum_p (t_p - y_p)^2$,

we want to determine

$\frac{\partial E}{\partial w^l_{m', n'}}$ in order to update the weights. $m'$ and $n'$ are the indices in the kernel matrix not be confused with its iterators. For example $w^1_{0,0} = -0.13$ in our example. We can also see that for an input image $H$x$K$ the output dimension after the convolutional layer will be

$(H-k_1+1)$x$(W-k_2+1)$.

In our case that would be $4$x$4$ as you showed. Let's calculate the error term. Each term found in the output space has been influenced by the kernel weights. The kernel weight $w^1_{0,0} = -0.13$ contributed to the output $x^1_{0,0} = 0.25$ and every single other output. Thus we express its contribution to the total error as

$\frac{\partial E}{\partial w^l_{m', n'}} = \sum_{i=0}^{H-k_1} \sum_{j=0}^{W-k_2} \frac{\partial E}{\partial x^l_{i, j}} \frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}}$.

This iterates across the entire output space, determines the error that output is contributing and then determines the contribution factor of the kernel weight with respect to that output.

Let us call the contribution to the error from the output space delta for simplicity and to keep track of the backpropagated error,

$\frac{\partial E}{\partial x^l_{i, j}} = \delta^l_{i,j}$.

The contribution from the weights

The convolution is defined as

$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$,

thus,

$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = \frac{\partial}{\partial w^l_{m', n'}} (\sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l)$.

By expanding the summation we end up observing that the derivative will only be non-zero when $m=m'$ and $n=n'$. We then get

$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = o^{l-1}_{i+m', j+n'}$.

Then back in our error term

$\frac{\partial E}{\partial w^l_{m', n'}} = \sum_{i=0}^{H-k_1} \sum_{j=0}^{W-k_2} \delta_{i,j}^l o^{l-1}_{i+m', j+n'}$.

Stochastic gradient descent

$w^{(t+1)} = w^{(t)} - \eta \frac{\partial E}{\partial w^l_{m', n'}}$

Let's calculate some of them

import numpy as np
from scipy import signal
o = np.array([(0.51, 0.9, 0.88, 0.84, 0.05), 
              (0.4, 0.62, 0.22, 0.59, 0.1), 
              (0.11, 0.2, 0.74, 0.33, 0.14), 
              (0.47, 0.01, 0.85, 0.7, 0.09),
              (0.76, 0.19, 0.72, 0.17, 0.57)])
d = np.array([(0, 0, 0.0686, 0), 
              (0, 0.0364, 0, 0), 
              (0, 0.0467, 0, 0), 
              (0, 0, 0, -0.0681)])

gradient = signal.convolve2d(np.rot90(np.rot90(y)), x, 'valid')

array([[ 0.044606, 0.094061], [ 0.011262, 0.068288]])

Now you can put that into the SGD equation in place of $\frac{\partial E}{\partial w}$.


Please let me know if theres errors in the derivation.

$\endgroup$
  • $\begingroup$ How $\frac{\partial E}{\partial w^l_{m', n'}}$ will look like in case my filter has multiple channels ? $\endgroup$ – koryakinp Oct 10 '18 at 14:11
  • 1
    $\begingroup$ gradient = signal.convolve2d(np.rot90(np.rot90(d)), o, 'valid') $\endgroup$ – Sun Bee May 7 at 20:02
  • $\begingroup$ I would like suggest to review this answer. In particular, the provided code in python might be checked $\endgroup$ – Duloren Jun 6 at 2:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.