I'm in the progress to learn, and understand different neural networks. I pretty much understand now feed-forward neural networks, and back-propagation of them, and now learning convolutional neural networks. I understand the forward-propagation of them, but having issues understanding their back-propagation. There is a very good resource explaining the convolutional layer, however, can't understand the back-propagation.

In my understanding, according the back-propagation algorithm of feed-forward neural networks/multi-layer perception, if I have the following input (its items as $i$), and filter (its items as $w$), giving the output (its items as $o$).

$$\begin{pmatrix}i_{1}^1 & i_{2}^1 & i_{3}^1\\\ i_{4}^1 & i_{5}^1 & i_{6}^1\\\ i_{7}^1 & i_{8}^1 & i_{9}^1\end{pmatrix} * \begin{pmatrix}w_1^1 & w_2^1\\\ w_3^1 & w_4^1\end{pmatrix} = \begin{pmatrix}o_1^1 & o_2^1\\\ o_3^1 & o_4^1\end{pmatrix}$$

So if we want to calculate for example how much $w_1^1$ affected the cost $C$, we need to know how much $w_1^1$ affected its corresponding output item $o_1^1$, and how much $o_1^1$ affected the cost $C$ which gives the following equation:

$$\frac{\partial C}{\partial w_1^1} = \frac{\partial o^1}{\partial w_1^1}\frac{\partial C}{\partial o^1}$$

Where in my thinking we have to think back how we get the output regarding to $w_1^1$ to calculate $\frac{\partial o^1}{\partial w_1^1}$.

To get $o_1^1$, we multiplied $w_1^1$ with $i_1^1$, to get $o_2^1$, multiplied $w_1^1$ with $i_2^1$, to get $o_3^1$, multiplied $w_1^1$ with $i_4^1$, to get $o_4^1$, multiplied $w_1^1$ with $i_5^1$.

To calculate $\frac{\partial C}{\partial o^1}$, it depends on how the output is connected with the next layer. If it is an another convolutional layer, then we have to calculate how each output item is connected to the next layers outputs, which will be their connecting weights.

So if we see an example, where we put a 2x2 filter on $o^1$, to get the final output $o^2$ (which will give a single output with 1x1 size):

$$\begin{pmatrix}o_1^1 & o_2^1\\\ o_3^1 & o_4^1\end{pmatrix} * \begin{pmatrix}w_1^2 & w_2^2\\\ w_3^2 & w_4^2\end{pmatrix} = \begin{pmatrix}o_1^2\end{pmatrix}$$

Where in my thinking the back-propagation for $w_1^2$ is:

$$\frac{\partial C}{\partial w_1^2} = \frac{\partial o^2}{\partial w_1^2}\frac{\partial C}{\partial o^2} = o_1^1 * 2(o^2_1 - y_1)$$,

and the back-propagation for $w_1^1$ is:

$$\frac{\partial C}{\partial w_1^1} = \frac{\partial o^1}{\partial w_1^1}\frac{\partial C}{\partial o^1}$$

Where: $$\frac{\partial o^1}{\partial w_1^1} = (i_1^1 + i_2^1 + i_4^1 + i_5^1)$$ And: $$\frac{\partial C}{\partial o^1} = \frac{\partial o_1^2}{\partial o_1^1}\frac{\partial C}{\partial o_1^2} + \frac{\partial o_1^2}{\partial o_2^1}\frac{\partial C}{\partial o_1^2} +\frac{\partial o_1^2}{\partial o_3^1}\frac{\partial C}{\partial o_1^2} +\frac{\partial o_1^2}{\partial o_4^1}\frac{\partial C}{\partial o_1^2}$$ So: $$\frac{\partial C}{\partial o^1} = w_1^2 * 2(o_1^2 - y_1) + w_2^2 * 2(o_1^2 - y_1) + w_3^2 * 2(o_1^2 - y_1) + w_4^2 * 2(o_1^2 - y_1)$$

Am I right? Because as I'm reading through the article above, it seems completely different.


1 Answer 1


Note that a CNN is a feed-forward neural network. Thus, if you understand how to perform backpropagation in feed-forward neural networks, you have it for CNNs.

A convolution layer can be understood as a fully connected layer, with the constraints that several edge weights are identical and many edge weights are set to 0.

You can also build a pooling layer in this way. For example an average pooling layer is nothing but a specific convolution layer, with fixed weights.

For max-pooling, use the fact that $\max\{x,y\} = \frac{x+y+|x-y|}{2}$.

  • $\begingroup$ Yes, when I realised that if I don't stick to the "pushing the filter on the image" thinking, it is a regular feed-forward network. But there is a big difference, and the question is to figure out if my thinking about it is right or not: in a feed-forward network a weight only connects a single neuron to get a single neuron. In conv, a weight is connected with multiple neurons to get a single neuron. And it will be the same weight, so in the back-propagation we have to take all connected neurons into account. $\endgroup$ Sep 8, 2020 at 11:31
  • $\begingroup$ 1.) In a feed-forward network, a weight is assigned to each edge that connects two neurons (and to each neuron if a bias is used). 2.) In a conv, the kernel matrix defines multiple weights that are applied to multiple neurons to obtain one new value at the kernel center. This is the same as using a fully connected layer, with appropriate edge weights. If transformed to a fully-connected layer, you must take the weight constraints into account! Edges outside the neighborhood have weights fixed to 0, and several edge weights are coupled together. You can do this with backprob. $\endgroup$ Sep 8, 2020 at 15:04
  • $\begingroup$ That's a bit hard to imagine, could you please use my equations in the question description, and fix the wrong ones to demonstrate what you mean? $\endgroup$ Sep 8, 2020 at 15:05
  • $\begingroup$ Do you know how to perform backprob for a fully connected layer, if you use weight sharing? If so, there is nothing new in using a conv layer (as you can easily transform a conv layer to a fully connected layer) $\endgroup$ Sep 8, 2020 at 15:10
  • $\begingroup$ Haven't heard about weight sharing in multilayer perception/fully connected layer $\endgroup$ Sep 8, 2020 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.