So i am trying to learn backpropagation of convolutional neural networks. A lot of articles only cover convolutions without a stride and a padding variable, so i decided to try it on my own.
For simplicity i decided to try correlation (no filter flipping).
Symbols:
$O$ = output
$W1$ = output width
$H1$ = output height
$I$ = input
$F$ = filter
$FS$ = filter size (filter width and height are the same and equal to $FS$)
So I defined the correlation as:
$$\label{1}\tag{1} O_{i,j} = \sum_{x = 0}^{FS-1}\sum_{y = 0}^{FS-1}I_{x+i*s-p, y+j*s-p}*F_{x,y}$$
Filter derivative:
$$\frac{\partial E}{\partial F_{i,j}} = \sum_{k=0}^{W1-1}\sum_{h=0}^{H1-1}\frac{\partial E}{\partial O_{k,h}}\frac{\partial O_{k,h}}{\partial F_{i,j}}$$
$$\frac{\partial O_{k,h}}{\partial F_{i,j}} = \frac{\partial }{\partial F_{i,j}}\left [ \sum_{x = 0}^{FS-1}\sum_{y = 0}^{FS-1}I_{x+k*s-p, y+h*s-p}*F_{x,y} \right ]$$
Derivative is non zero only when $x = i$ and $y = j$, therefore:
$$ \frac{\partial O_{k,h}}{\partial F_{i,j}} = I_{i+k*s-p, j+h*s-p} $$
and
$$\label{2}\tag{2} \frac{\partial E}{\partial F_{i,j}} = \sum_{k=0}^{W1-1}\sum_{h=0}^{H1-1}\frac{\partial E}{\partial O_{k,h}}I_{i+k*s-p, j+h*s-p}$$
Code attempt:
So i wanted to try if my calculations were correct, so i wrote this simple correlation and backpropagation in javascript: https://gist.github.com/jakic12/414ad450d9c1222e58d8e09c6b92cebb
First the forward propagation (correlation) - equation (1)
/**
* correlate an array `a` with a filter `f`
* @param{*} a the input array
* @param{*} f the filter
* @param{*} s stride
* @param{*} p padding
*/
function corre(a, f, s, p){
let outY = parseInt((a.length - f.length + 2 * p)/s + 1)
let outX = parseInt((a[0].length - f[0].length + 2 * p)/s + 1)
return new Array(outY).fill(0).map((_, y) =>
new Array(outX).fill(0).map((__, x) => {
let sum = 0;
for(let j = 0; j < f.length; j++){
for(let i = 0; i < f[j].length; i++){
if(a[y + j * s - p] && a[y + j * s - p][x + i * s - p])
sum += a[y + j * s - p][x + i * s - p] * f[y][x]
}
}
return sum
}
)
)
}
Then the error calculation (Mean squared error) and the partial derivatives with respect to the output layer ($\frac{\partial E}{\partial O_{k,h}}$)
/**
* Calculate the error and the partial derivatives with respect to the output layer
* @param{*} actual the actual output
* @param{*} exp the expected output
*/
function getError(actual, exp){
let err = 0
let out = new Array(actual.length).fill(0).map(() => new Array(actual[0].length))
for(let i = 0; i < out.length; i++){
for(let j = 0; j < out[i].length; j++){
out[i][j] = actual[i][j] - exp[i][j]
err += Math.pow(exp[i][j] - actual[i][j],2)
}
}
return { dO:out, err:err/2 }
}
Then calculating the derivatives with respect to the filter - equation (2)
/**
* backpropagate the correlation with given derivatives of the next layer
* @param{*} FS filter size
* @param{*} dO derivative with respect to the output of the correlation
* @param{*} input the input of the correlation
* @param{*} s stride
* @param{*} p pad
*/
function backpropFilter(FS, dO, input, s, p){
return new Array(FS).fill(0).map((_, j) =>
new Array(FS).fill(0).map((__, i) => {
let sum = 0
for(let h = 0; h < dO.length; h++){
for(let k = 0; k < dO[h].length; k++){
if(j+h*s-p >= 0 && j+h*s-p < input.length && i+k*s-p >= 0 && i+k*s-p < input[0].length)
sum += dO[h][k] * input[j+h*s-p][i+k*s-p]
}
}
return sum
}
)
)
}
The problem
The problem is, that altho the network first starts descending, it then quickly starts going up and never stops:
I am sorry, because i am not the best at derivatives, so i don't know where my problem is.
Help would be appreciated, thanks!
