- Gradient is derivative of several variables.
- I can't understand why is the gradient of a ReLU for X>0 is 1 ? and 0 for x < 0 ?
I tried to search for proof and examples but didn't found any good examples.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
The ReLU function is defined as follows: $f(x) = max(0, x)$, meaning that the output of the function is maximum between the input value and zero. This can also be written as follows:
$ f(x) = \begin{cases} 0 & \text{if } x \leq 0, \\ x & \text{if } x \gt 0 \end{cases} $
If we then simply take the derivate of the two outputs with respect to $x$ we get the gradient for input values below zero and value greater than or equal to zero.
$ f'(x) = \begin{cases} 0 & \text{if } x \leq 0, \\ 1 & \text{if } x \gt 0 \end{cases} $
Therefore the gradient of the ReLU function is zero for values up to and including zero and 1 for positive values.
-
2$\begingroup$ The derivative is not defined at zero. You can take the derivative piecewise inside intervals, but not at their endpoints; and a quick look at the graph of ReLU makes it clear that the derivative doesn't exist at 0. (For the purposes of back-propagation, a "sub-gradient" is sufficient, and zero works for that purpose.) $\endgroup$– Ben Reiniger ♦Jun 1, 2021 at 14:24