1
$\begingroup$

I am trying to derive the backpropagation for a single convolutional layer (padding layer is being implemented separately, so no padding argument for the convolutional layer). This layer is given $\frac{\partial L}{\partial O}$ from the last backward propagation.

Layer input: $X$, shape = $n_{ch} \times h \times w$

Layer output: $O$, shape: $n_f \times h_{new} \times w_{new}$

Layer weight: $W$, shape: $n_f \times n_{ch} \times f_r \times f_c$

Layer bias: $b$, shape: $n_f \times 1 \times 1 \times 1$

(where $n_{ch}$ is the number of channels, $f_r$ and $f_c$ are the number of rows and columns of a filter respectively, usually $f_r$ = $f_c$)

Attempt:

I have that the output corresponding to a filter $f$ is:

$O_{f,i,j}$ = ($\sum_{ch=1}^{n_{ch}}\sum_{r=1}^{f_r}\sum_{c=1}^{f_c}W_{f, ch, r, c} * X_{ch, (i-1)s+r, (j-1)s+c}) + b_f$

where $s$ is the stride.

To calculate $\frac{\partial L}{\partial X}$ to pass to the layer behind, and $\frac{\partial L}{\partial W}$, $\frac{\partial L}{\partial b}$ to update the parameters, I try using $\frac{\partial L}{\partial O}$:

$\frac{\partial L}{\partial X_{ch,p,q}} = \sum_{f=1}^{n_f}\sum_{i=1}^{h_{new}}\sum_{j=1}^{w_{new}}\frac{\partial L}{\partial O_{f,i,j}}\frac{\partial O_{f,i,j}}{\partial X_{ch,p,q}}$

$\frac{\partial O_{f,i,j}}{\partial X_{ch,p,q}} = \begin{cases} W_{f,ch,p-(i-1)s,q-(j-1)s}, \quad 1<=p-(i-1)s<=f_r \cap 1<=q-(j-1)s<=f_c\\ 0, \quad otherwise \end{cases}$

$\boxed{\frac{\partial L}{\partial X_{ch,p,q}} = \sum_{f=1}^{n_f}\sum_{i=i_{lower}}^{i_{upper}}\sum_{j=j_{lower}}^{j_{upper}}\frac{\partial L}{\partial O_{f,i,j}}W_{f,ch,p-(i-1)s,q-(j-1)s}}$

where $i_{lower} = \lfloor{\frac{p-f_r}{s} + 1}\rfloor, i_{upper} = \lfloor{\frac{p-1}{s} + 1}\rfloor, j_{lower} = \lfloor{\frac{q-f_c}{s} + 1}\rfloor, j_{upper} = \lfloor{\frac{q-1}{s} + 1}\rfloor$ (calculated by solving for $i$, $j$ in the inequalities).

Similarly I found that:

$\frac{\partial O_{f_1,i,j}}{\partial W_{f_2,ch,r,c}} = \begin{cases} X_{ch,(i-1)s+r,(j-1)s+c}, \quad f_1 = f_2\\ 0, \quad otherwise \end{cases}$

$\boxed{\frac{\partial L}{\partial W_{f,ch,r,c}} = \sum_{i,j}\frac{\partial L}{\partial O_{f,i,j}}X_{ch,(i-1)s+r,(j-1)s+c}}$

$\boxed{\frac{\partial L}{\partial b_f} = \sum_{i,j}\frac{\partial L}{\partial O_{f,i,j}}}$

Is this approach correct, and how would you even begin to vectorize this for implementation?

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.