ReLU is an activation function defined as $h = \max(0, a)$ where $a = Wx + b$.

Normally, we train neural networks with first-order methods such as SGD, Adam, RMSprop, Adadelta, or Adagrad. Backpropagation in first-order methods requires first-order derivative. Hence $x$ is derived to $1$.

But if we use second-order methods, would ReLU's derivative be $0$? Because $x$ is derived to $1$ and is derived again to $0$. Would it be an error? For example, with Newton's method, you'll be dividing by $0$. (I don't really understand Hessian-free optimization, yet. IIRC, it's a matter of using an approximate Hessian instead of the real one).

What is the effect of this $h''=0$? Can we still train the neural network with ReLU with second-order methods? Or would it be non-trainable/error (nan/infinity)?

For clarity, this is ReLU as $f(x)$:

$f(x) =$ \begin{array}{rcl} 0 & \mbox{for} & x < 0\\ x & \mbox{for} & x \ge 0\end{array}

$f'(x) =$ \begin{array}{rcl} 0 & \mbox{for} & x < 0\\ 1 & \mbox{for} & x \ge 0\end{array}

$f''(x) = 0$


1 Answer 1


Yes the ReLU second order derivative is 0. Technically, neither $\frac{dy}{dx}$ nor $\frac{d^2y}{dx^2}$ are defined at $x=0$, but we ignore that - in practice an exact $x=0$ is rare and not especially meaningful, so this is not a problem. Newton's method does not work on the ReLU transfer function because it has no stationary points. It also doesn't work meaningfully on most other common transfer functions though - they cannot be minimised or maximised for finite inputs.

When you combine multiple ReLU functions with layers of matrix multiplications in a structure such as a neural network, and wish to minimise an objective function, the picture is more complicated. This combination does have stationary points. Even a single ReLU neuron and a mean square error objective will have different enough behaviour such that the second-order derivative of the single weight will vary and is not guaranteed to be 0.

Nonlinearities when multiple layers combine is what creates a more interesting optimisation surface. This also means that it is harder to calculate useful second-order partial derivatives (or Hessian matrix), it is not just a matter of taking second order derivatives of the transfer functions.

The fact that $\frac{d^2y}{dx^2} = 0$ for the transfer function will make some terms zero in the matrix (for the second order effect from same neuron activation), but the majority of terms in the Hessian are of the form $\frac{\partial^2E}{\partial x_i\partial x_j}$ where E is the objective and $x_i$, $x_j$ are different parameters of the neural network. A fully-realised Hessian matrix will have $N^2$ terms where $N$ is number of parameters - with large neural networks having upwards of 1 million parameters, even with a simple calculation process and many terms being 0 (e.g. w.r.t. 2 weights in same layer) this may not be feasible to compute.

There are techniques to estimate effects of second-order derivatives used in some neural network optimisers. RMSProp can be viewed as roughly estimating second-order effects, for example. The "Hessian-free" optimisers more explicitly calculate the impact of this matrix.

  • $\begingroup$ So it will be just fine? Not error (nan/infinity)? $\endgroup$ Commented Jul 12, 2016 at 22:49
  • $\begingroup$ I heard about Hessian-free optimization, I thought that's just about using an approximated Hessian instead of the real one? $\endgroup$ Commented Jul 12, 2016 at 22:50
  • 1
    $\begingroup$ Yes I think most techniques estimate Hessian rather than attempt to robustly calculate it. I don't know why, but would assume because it is hard or impossible to construct the backpropagation as second order effects will entangle terms e.g. $\frac{\partial^2}{\partial x\partial y}$ where x and y are params from different layers. $\endgroup$ Commented Jul 13, 2016 at 6:35
  • 1
    $\begingroup$ @rilut: I have updated the answer, as I think your confusion stems from the fact that Newton's method would not work on the ReLU transfer function. You are correct about that. But once you combine ReLU with an objective function (even something as simple as a single neuron and mean square error objective), it will start to work. $\endgroup$ Commented Jul 13, 2016 at 9:20
  • $\begingroup$ Awesome. Just awesome. $\endgroup$ Commented Jul 13, 2016 at 9:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.