# How can the ReLU function lead to convergence?

The gradient descent algorithm is based on the fact that the gradient decreases as we move towards the optimum point. However, in the activations by the ReLU function, the gradient will be constant and will not change as the input changes.

I am unclear how this will finally lead to convergence. I would be grateful if you could explain this with mathematical derivations. Thanks

ReLU isn’t the only function being applied to the data to produce an output. Each layer is a linear transformation of the last, followed by RELU. Even if everything is negative and the ReLU doesn’t contribute to the gradient, the gradient with respect to all of the model weights will almost certainly be nonzero, unless you have converged to a critical point.

Short answer: When updating the weights (or parameters) of your machine learning architecture, you move along the gradient of the loss function applied to the empirical data and the data that your model predicts. This gradient can (and hopefully will, but doesn't have to) decrease as the number of epochs increases, so training will go on just fine.

Example. Consider one of the simplest "machine learning" problems: Given a set of points $$S=\{(x_1,y_1),(x_2,y_2),\dots, (x_N, y_N)\}\subset \mathbb R^2, N\in\mathbb N,$$

we want to find the best-fit line to these points, i.e. we want to find $$m,b\in\mathbb R$$ such that

$$f_{m,b}:\mathbb R\to\mathbb R, f_{m,b}(x)=mx+b$$

$$\mathcal L(m,b;S)=\sum_{k=1}^N (f_{m,b}(x_k)-y_k)^2.$$
Now, note that, for fixed $$S$$, $$\mathcal L$$ is a convex function (actually I haven't checked this, let me know if I am mistaken here) and, as you can check as an exercise, if there exists a minimizer $$(m^*,b^*)$$ of $$\mathcal L$$, then "gradient descent" will converge towards this minimizer (note that there is an unfortunate bug in my formulation which causes minimizers to not always exist: This bug occurs when the best fit would be a vertical line, which can't be expressed as $$y=mx+b$$).
Note that the same is also true if you take for instance $$g_{m,b}=\operatorname{Relu}(mx+b)$$, even though the gradient of both $$\operatorname{Relu}$$ and $$mx+b$$ does not have to converge to $$0$$ as we converge to the minimizer.