• Gradient is derivative of several variables.
  • I can't understand why is the gradient of a ReLU for X>0 is 1 ? and 0 for x < 0 ?

I tried to search for proof and examples but didn't found any good examples.


The ReLU function is defined as follows: $f(x) = max(0, x)$, meaning that the output of the function is maximum between the input value and zero. This can also be written as follows:

$ f(x) = \begin{cases} 0 & \text{if } x \leq 0, \\ x & \text{if } x \gt 0 \end{cases} $

If we then simply take the derivate of the two outputs with respect to $x$ we get the gradient for input values below zero and value greater than or equal to zero.

$ f'(x) = \begin{cases} 0 & \text{if } x \leq 0, \\ 1 & \text{if } x \gt 0 \end{cases} $

Therefore the gradient of the ReLU function is zero for values up to and including zero and 1 for positive values.

  • 2
    $\begingroup$ The derivative is not defined at zero. You can take the derivative piecewise inside intervals, but not at their endpoints; and a quick look at the graph of ReLU makes it clear that the derivative doesn't exist at 0. (For the purposes of back-propagation, a "sub-gradient" is sufficient, and zero works for that purpose.) $\endgroup$ Jun 1 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.