Imagine I have the following layers in a CNN:
Conv-layer 1 (without reLU) [3 filters @ 1x3x3] => ReLU-layer 1 => Maxpooling-layer 1 [2x2]
Conv-layer 2 (without reLU) [10 filters @ 3x4x4] => ReLU-layer 2 => Maxpooling-layer 2 [2x2]
What I know is that, when "deriving" the Maxpooling-layer, the maximum value of that pooling has the value of 1 (because that value is the only one that is affecting the result of the CNN), and the other values take the value of 0.
After this step, I have the reLU-layer, where the derivative of the reLU-layer is 0 when x <= 0, and 1 when x > 0.
And in the last step, I have the Conv-layer, but I don't know how to calculate the derivative. How do I calculate the whole delta for that specific weight?
If there is a way to calculate the gradients of those weights using matrix notation, please let me know.
The matrix notation that I use is the following: [number of filters - depth - rows - columns]. For example, when I say [3 filters - 1 x 3 x 3] I mean that I have 3 filters with depth 1, 3 rows and 3 columns (depth is "equivalent" to "channels").
Steps in Conv layer, using my notation
Imagine I have an input X with shape [depth:1, rows: 28, columns: 28]
Then, I have the conv-layer, where I have 3 filters (you can call the filter as W) with shape [depth:1, rows:3, columns:3]
So, what I do is, using 1 filter a time, convolve input X , so the resulting matrix has the following shape => [depth:1, rows: 26, columns: 26] (this is the result of convolving 1 filter). So, when I do the convolution using the 3 filters, the resulting matrix is shaped [depth:3, rows: 26, columns: 26].
After this, I apply ReLU to each value of the resulting matrix (the 3x26x26 matrix), and then apply non-overlapping maxpooling with a window of 2 rows x 2 columns.