I am trying to derive the backpropagation for a single convolutional layer (padding layer is being implemented separately, so no padding argument for the convolutional layer). This layer is given $\frac{\partial L}{\partial O}$ from the last backward propagation.

Layer input: $X$, shape = $n_{ch} \times h \times w$

Layer output: $O$, shape: $n_f \times h_{new} \times w_{new}$

Layer weight: $W$, shape: $n_f \times n_{ch} \times f_r \times f_c$

Layer bias: $b$, shape: $n_f \times 1 \times 1 \times 1$

(where $n_{ch}$ is the number of channels, $f_r$ and $f_c$ are the number of rows and columns of a filter respectively, usually $f_r$ = $f_c$)


I have that the output corresponding to a filter $f$ is:

$O_{f,i,j}$ = ($\sum_{ch=1}^{n_{ch}}\sum_{r=1}^{f_r}\sum_{c=1}^{f_c}W_{f, ch, r, c} * X_{ch, (i-1)s+r, (j-1)s+c}) + b_f$

where $s$ is the stride.

To calculate $\frac{\partial L}{\partial X}$ to pass to the layer behind, and $\frac{\partial L}{\partial W}$, $\frac{\partial L}{\partial b}$ to update the parameters, I try using $\frac{\partial L}{\partial O}$:

$\frac{\partial L}{\partial X_{ch,p,q}} = \sum_{f=1}^{n_f}\sum_{i=1}^{h_{new}}\sum_{j=1}^{w_{new}}\frac{\partial L}{\partial O_{f,i,j}}\frac{\partial O_{f,i,j}}{\partial X_{ch,p,q}}$

$\frac{\partial O_{f,i,j}}{\partial X_{ch,p,q}} = \begin{cases} W_{f,ch,p-(i-1)s,q-(j-1)s}, \quad 1<=p-(i-1)s<=f_r \cap 1<=q-(j-1)s<=f_c\\ 0, \quad otherwise \end{cases}$

$\boxed{\frac{\partial L}{\partial X_{ch,p,q}} = \sum_{f=1}^{n_f}\sum_{i=i_{lower}}^{i_{upper}}\sum_{j=j_{lower}}^{j_{upper}}\frac{\partial L}{\partial O_{f,i,j}}W_{f,ch,p-(i-1)s,q-(j-1)s}}$

where $i_{lower} = \lfloor{\frac{p-f_r}{s} + 1}\rfloor, i_{upper} = \lfloor{\frac{p-1}{s} + 1}\rfloor, j_{lower} = \lfloor{\frac{q-f_c}{s} + 1}\rfloor, j_{upper} = \lfloor{\frac{q-1}{s} + 1}\rfloor$ (calculated by solving for $i$, $j$ in the inequalities).

Similarly I found that:

$\frac{\partial O_{f_1,i,j}}{\partial W_{f_2,ch,r,c}} = \begin{cases} X_{ch,(i-1)s+r,(j-1)s+c}, \quad f_1 = f_2\\ 0, \quad otherwise \end{cases}$

$\boxed{\frac{\partial L}{\partial W_{f,ch,r,c}} = \sum_{i,j}\frac{\partial L}{\partial O_{f,i,j}}X_{ch,(i-1)s+r,(j-1)s+c}}$

$\boxed{\frac{\partial L}{\partial b_f} = \sum_{i,j}\frac{\partial L}{\partial O_{f,i,j}}}$

Is this approach correct, and how would you even begin to vectorize this for implementation?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.