# Backpropagation derivation for a convolutional layer

I am trying to derive the backpropagation for a single convolutional layer (padding layer is being implemented separately, so no padding argument for the convolutional layer). This layer is given $$\frac{\partial L}{\partial O}$$ from the last backward propagation.

Layer input: $$X$$, shape = $$n_{ch} \times h \times w$$

Layer output: $$O$$, shape: $$n_f \times h_{new} \times w_{new}$$

Layer weight: $$W$$, shape: $$n_f \times n_{ch} \times f_r \times f_c$$

Layer bias: $$b$$, shape: $$n_f \times 1 \times 1 \times 1$$

(where $$n_{ch}$$ is the number of channels, $$f_r$$ and $$f_c$$ are the number of rows and columns of a filter respectively, usually $$f_r$$ = $$f_c$$)

Attempt:

I have that the output corresponding to a filter $$f$$ is:

$$O_{f,i,j}$$ = ($$\sum_{ch=1}^{n_{ch}}\sum_{r=1}^{f_r}\sum_{c=1}^{f_c}W_{f, ch, r, c} * X_{ch, (i-1)s+r, (j-1)s+c}) + b_f$$

where $$s$$ is the stride.

To calculate $$\frac{\partial L}{\partial X}$$ to pass to the layer behind, and $$\frac{\partial L}{\partial W}$$, $$\frac{\partial L}{\partial b}$$ to update the parameters, I try using $$\frac{\partial L}{\partial O}$$:

$$\frac{\partial L}{\partial X_{ch,p,q}} = \sum_{f=1}^{n_f}\sum_{i=1}^{h_{new}}\sum_{j=1}^{w_{new}}\frac{\partial L}{\partial O_{f,i,j}}\frac{\partial O_{f,i,j}}{\partial X_{ch,p,q}}$$

$$\frac{\partial O_{f,i,j}}{\partial X_{ch,p,q}} = \begin{cases} W_{f,ch,p-(i-1)s,q-(j-1)s}, \quad 1<=p-(i-1)s<=f_r \cap 1<=q-(j-1)s<=f_c\\ 0, \quad otherwise \end{cases}$$

$$\boxed{\frac{\partial L}{\partial X_{ch,p,q}} = \sum_{f=1}^{n_f}\sum_{i=i_{lower}}^{i_{upper}}\sum_{j=j_{lower}}^{j_{upper}}\frac{\partial L}{\partial O_{f,i,j}}W_{f,ch,p-(i-1)s,q-(j-1)s}}$$

where $$i_{lower} = \lfloor{\frac{p-f_r}{s} + 1}\rfloor, i_{upper} = \lfloor{\frac{p-1}{s} + 1}\rfloor, j_{lower} = \lfloor{\frac{q-f_c}{s} + 1}\rfloor, j_{upper} = \lfloor{\frac{q-1}{s} + 1}\rfloor$$ (calculated by solving for $$i$$, $$j$$ in the inequalities).

Similarly I found that:

$$\frac{\partial O_{f_1,i,j}}{\partial W_{f_2,ch,r,c}} = \begin{cases} X_{ch,(i-1)s+r,(j-1)s+c}, \quad f_1 = f_2\\ 0, \quad otherwise \end{cases}$$

$$\boxed{\frac{\partial L}{\partial W_{f,ch,r,c}} = \sum_{i,j}\frac{\partial L}{\partial O_{f,i,j}}X_{ch,(i-1)s+r,(j-1)s+c}}$$

$$\boxed{\frac{\partial L}{\partial b_f} = \sum_{i,j}\frac{\partial L}{\partial O_{f,i,j}}}$$

Is this approach correct, and how would you even begin to vectorize this for implementation?

• This article might help you in some steps, as it explains how to convert CNN mathematical formulas into code: analyticsvidhya.com/blog/2020/02/… Sep 15 at 8:45