I'm trying to implement a CNN, as part of an academic project to learn how it works. The project is a SRCNN: a convolutional neural network that increase the resolution of images.

Following this review Review: SRCNN (Super Resolution), the network includes 2 convolution layers with ReLu activation and 1 convolution layer without ReLu activation. Feed forwarding is quite simple. I'm having trouble on the backpropagation though.

Following this article CNNs, Part 2: Training a Convolutional Neural Network, I got the following derivative to update filters weights :

$$ \frac{\partial L}{\partial filter(x,y)}​​=\sum_{i=x}^{W-S+x}\sum_{j=y}^{H-S+y}\frac{\partial L}{\partial out(i,j)}*image(i+x,j+y) $$

where $L$ is the loss, $filter(x,y)$ the weight at the position $(x,y)$, $W$ the width of the image, $H$ the height of the image, $S$ the size of the filter.

But it does not take the activation function into account. It should work for the convolution layer without activation. But I can't see how to integrate ReLu derivative in this. And I didn't find any article that explains it clearly.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.