1
$\begingroup$

So i am trying to learn backpropagation of convolutional neural networks. A lot of articles only cover convolutions without a stride and a padding variable, so i decided to try it on my own. For simplicity i decided to try correlation (no filter flipping).
Symbols:

$O$ = output
$W1$ = output width
$H1$ = output height
$I$ = input
$F$ = filter
$FS$ = filter size (filter width and height are the same and equal to $FS$)

So I defined the correlation as:
$$\label{1}\tag{1} O_{i,j} = \sum_{x = 0}^{FS-1}\sum_{y = 0}^{FS-1}I_{x+i*s-p, y+j*s-p}*F_{x,y}$$

Filter derivative:

$$\frac{\partial E}{\partial F_{i,j}} = \sum_{k=0}^{W1-1}\sum_{h=0}^{H1-1}\frac{\partial E}{\partial O_{k,h}}\frac{\partial O_{k,h}}{\partial F_{i,j}}$$ $$\frac{\partial O_{k,h}}{\partial F_{i,j}} = \frac{\partial }{\partial F_{i,j}}\left [ \sum_{x = 0}^{FS-1}\sum_{y = 0}^{FS-1}I_{x+k*s-p, y+h*s-p}*F_{x,y} \right ]$$ Derivative is non zero only when $x = i$ and $y = j$, therefore:
$$ \frac{\partial O_{k,h}}{\partial F_{i,j}} = I_{i+k*s-p, j+h*s-p} $$ and
$$\label{2}\tag{2} \frac{\partial E}{\partial F_{i,j}} = \sum_{k=0}^{W1-1}\sum_{h=0}^{H1-1}\frac{\partial E}{\partial O_{k,h}}I_{i+k*s-p, j+h*s-p}$$

Code attempt:

So i wanted to try if my calculations were correct, so i wrote this simple correlation and backpropagation in javascript: https://gist.github.com/jakic12/414ad450d9c1222e58d8e09c6b92cebb

First the forward propagation (correlation) - equation (1)

/**
 * correlate an array `a` with a filter `f`
 * @param{*} a the input array
 * @param{*} f the filter
 * @param{*} s stride
 * @param{*} p padding
 */
function corre(a, f, s, p){
    let outY = parseInt((a.length - f.length + 2 * p)/s + 1)
    let outX = parseInt((a[0].length - f[0].length + 2 * p)/s + 1)

    return new Array(outY).fill(0).map((_, y) => 
        new Array(outX).fill(0).map((__, x) => {
            let sum = 0;
            for(let j = 0; j < f.length; j++){
                for(let i = 0; i < f[j].length; i++){
                    if(a[y + j * s - p] && a[y + j * s - p][x + i * s - p])
                        sum += a[y + j * s - p][x + i * s - p] * f[y][x]
                }
            }
            return sum
            }
        )
    )
}

Then the error calculation (Mean squared error) and the partial derivatives with respect to the output layer ($\frac{\partial E}{\partial O_{k,h}}$)

/**
 * Calculate the error and the partial derivatives with respect to the output layer
 * @param{*} actual the actual output
 * @param{*} exp the expected output
 */
function getError(actual, exp){
    let err = 0
    let out = new Array(actual.length).fill(0).map(() => new Array(actual[0].length))

    for(let i = 0; i < out.length; i++){
        for(let j = 0; j < out[i].length; j++){
            out[i][j] = actual[i][j] - exp[i][j]
            err += Math.pow(exp[i][j] - actual[i][j],2)
        }
    }
    return { dO:out, err:err/2 }
}

Then calculating the derivatives with respect to the filter - equation (2)

/**
 * backpropagate the correlation with given derivatives of the next layer
 * @param{*} FS filter size
 * @param{*} dO derivative with respect to the output of the correlation
 * @param{*} input the input of the correlation
 * @param{*} s stride
 * @param{*} p  pad
 */
function backpropFilter(FS, dO, input, s, p){
    return new Array(FS).fill(0).map((_, j) =>
        new Array(FS).fill(0).map((__, i) => {
            let sum = 0
    
            for(let h = 0; h < dO.length; h++){
                for(let k = 0; k < dO[h].length; k++){          
                    if(j+h*s-p >= 0 && j+h*s-p < input.length && i+k*s-p >= 0 && i+k*s-p < input[0].length)
                        sum += dO[h][k] * input[j+h*s-p][i+k*s-p]
                }
            }
            return sum
            }
        )
    )
}

The problem

The problem is, that altho the network first starts descending, it then quickly starts going up and never stops: Error graph when learning with 10000 iterations

I am sorry, because i am not the best at derivatives, so i don't know where my problem is.
Help would be appreciated, thanks!

$\endgroup$

1 Answer 1

0
$\begingroup$

There is a typo in the code:

/**
 * correlate an array `a` with a filter `f`
 * @param{*} a the input array
 * @param{*} f the filter
 * @param{*} s stride
 * @param{*} p padding
 */
function corre(a, f, s, p){
    let outY = parseInt((a.length - f.length + 2 * p)/s + 1)
    let outX = parseInt((a[0].length - f[0].length + 2 * p)/s + 1)

    return new Array(outY).fill(0).map((_, y) => 
        new Array(outX).fill(0).map((__, x) => {
            let sum = 0;
            for(let j = 0; j < f.length; j++){
                for(let i = 0; i < f[j].length; i++){
                    if(a[y + j * s - p] && a[y + j * s - p][x + i * s - p])
                        sum += a[y + j * s - p][x + i * s - p] * f[y][x]
                }
            }
            return sum
            }
        )
    )
}
sum += a[y + j * s - p][x + i * s - p] * f[y][x]
                                           ^  ^

should be

sum += a[y + j * s - p][x + i * s - p] * f[j][i]
                                           ^  ^
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.