Relu only passes positive values, so when we calculate the gradients for this layer, we will only get positive gradients. The gradient for the filter weights of this layer is the convolution of the input image and the derivative of the loss function (which, as I said, is positive). This means that the gradient is also positive, which means that the filter weights are always decreasing. Something seems off.

I have tried to use leaky_relu, but filter weights are still decreasing (just with the lower rate)


3 Answers 3


Welcome to the DataScience stack exchange. ReLu is not "correct" or "incorrect" but it is just one of several popular choices for a nonlinearity in neural networks.

It sounds like either you're misunderstanding something , and/or you need to clarify your question. But I'll do my best to clarify some things.

gradient for the filter weights of this layer

ReLu layers don't have weights, but you probably mean the weights of some layer prior to the ReLu.

This means that the gradient is also positive

It's true that the gradient of a ReLu layer with respect to its input is always positive (equal to one) or zero.

which means that the filter weights are always decreasing.

Sounds like you're talking about using backprop to update weights "through" a ReLu layer, and that statement is not correct. Filter weights can decrease because during backprop, one computes the gradient of a loss function with respect to its weights. That is not the same as the gradient of the ReLu with respect to its inputs.

  • $\begingroup$ youtu.be/… . In my case dL/dz - is gradient after Relu layer, right? $\endgroup$
    – Tima
    Commented Dec 2, 2023 at 19:32
  • $\begingroup$ I didn't see anything about ReLu in that video. $\endgroup$
    – bogovicj
    Commented Dec 2, 2023 at 21:59
  • $\begingroup$ Yes, this video is not about Relu, but there is always an activation function after convolutional layer - so, i used Relu. To find dL / dz (which is in video), we have to compute all gradients for next layers. That what i mean - derivative of activation func (Relu) is already in dL / dZ. Maybe, i can't explain clearly my problem due to language barrier. $\endgroup$
    – Tima
    Commented Dec 2, 2023 at 22:10

I think you're confused about a lot of concepts here.

First an empirical example:

import torch
import torch.nn as nn

layer = nn.Conv2d(8, 16, 3, padding='same')
relu = nn.ReLU()

x = torch.randn(1, 8, 128, 128)

y = relu(layer(x)).mean()


You can manually inspect the gradients of your conv layer and verify there are both positive and negative values. If your model weights are decreasing in magnitude, it is either because this helps the model perform better, or something like weight decay is forcing the behavior.

When we compute grads via chain rule/backprop, the gradient of your conv layer is the gradient of that specific conv layer multiplied by the gradient of the next layer in the model.

The sign of the relu gradient does not determine the sign of the conv weight gradient. The relu gradient is 1 for x>0, 0 for x<0. This means that the relu gradient masks the gradients from the next layer of the model.

Models are trained with multiple items in a batch. If the relu zeros an element in one batch item, it may not be zeroed in another batch item. The overall update of the model depends on all the gradients from all batch items, so having zero gradient on specific elements does not prevent the model from learning.

  • $\begingroup$ Thanks, i have solved my problem $\endgroup$
    – Tima
    Commented Dec 3, 2023 at 10:52
  • $\begingroup$ What if batch size for example = 10. One image as input and one filter in layer. So what is the gradient for filter? Mean value of the gradient of 10 objects or smth else? $\endgroup$
    – Tima
    Commented Dec 3, 2023 at 11:46

Thanks for your answer. I watched this nice guy to understand the concept of CNN and its backprop. I watched more carefully this - video and found out the answer to my problem. He said that there is a dot product multiplication between dL / dC and dC / dZ. My issue was - i used Hadamard product multiplication between dL / dC and dC / dZ. I am not really understand why we use here a dot product for Relu backprop, while Relu forward prop is element-wise enter image description here

enter image description here

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.