# Differentiable vs Non Differentiable loss function in ML

ML Question: What is a differentiable loss function and why does it matter?

For example, for a given Input training set; the loss function is: $$L(y,F(x))$$

Is this differentiable? How? And why does it matter?

In Layman's terms:

1. A loss function may be differentiable or else, depending on its exact form.

2. We like differentiable function because it allows the use of gradient-based optimization methods e.g. SGD. On the other hand, optimization methods for non-differentiable functions e.g. genetic algorithm are usually harder/takes longer to compute.

3. While training a model, we search for the minimum of the loss function, during which the optimization method is computed for a lot of times; thus a differentiable loss function is preferred for efficiency.

A loss function is differentiable if you can compute its derivative with respect to the model parameters.

In your example, there is not enough information to say if the loss function is differentiable, because we don't know how $$L$$ and $$F$$ are defined.

An example of differentiable loss function is the mean squared error, that is $$\mathbf{L} = \frac{1}{n}\sum_{i=1}^n(y_i - F(x_i))^2$$, where $$F$$ is our model (e.g. a neural network), which must also be differentiable.

Differentiable loss functions matter for gradient-based optimization approaches, like gradient descent for neural networks, because to be able to apply them you need the loss function (and the model) to be differentiable.

• @noe I understand the idea behind a differentiable loss function for gradient based optimization as we're searching for the minimum loss function value. What do you mean that the model must be differentiable as well? And for a LLM with say 100bn paramters, what does it mean to compute derivative w.r.t the parameters?
– kms
Commented Jan 29, 2023 at 7:26
• The model needs to be differentiable because the gradient back-propagates through the model and, for that, you need the derivative of each layer/computational node. When you apply back-propagation, you need to compute the gradient with respect to the parameters. The key is that you express the gradient locally at each layer/computational node with respect to the back-propagated gradient.
– noe
Commented Jan 29, 2023 at 9:34
• Hi, How would I know if a loss function is differentiable? aside from the definition of a differentiable function. How would you gauge it to be differentiable and if not make an approximation to it please. Thanks Commented Apr 2, 2023 at 11:47

A function $$f: A \to \mathbb{R}$$, with $$A \subset \mathbb{R}$$, where $$\mathbb{R}$$ is the set of real numbers, is differentiable at $$a \in A$$ if its derivative:

$$f'(a) = \lim_{h \to 0}\frac{f(a+h) - f(a)}{f(h)}$$

exists, which implies that $$f$$ is continuous at $$a$$. The function $$f$$ is said to be differentiable on $$A$$ if it is differentiable at every point of $$A$$.

Differentiable loss functions are the bread and butter in training neural networks.

Historically, artificial intelligence (AI), and thus machine learning, started in 1956 with the proposal of McCharty et al. for a two-month, 10-person summer conference at Dartmouth College. Without going too deep with the chronological milestones, it is important to stress that many promises by AI researchers of future successes failed. The latter started what is called an AI winter. The return of neural networks came in 1986 thanks to the influential work on the backpropagation algorithm by Rumelhart, Hinton, and Williams.

To understand backpropagation, you must comprehend what partial derivatives are and what the gradient of a function is. Let $$A \subset \mathbb{R}^n$$ be an open set on $$\mathbb{R}^n$$, $$f: A \to \mathbb{R}$$ a function of $$n$$ variables (i.e., $$f(x_1,\ldots,x_n)$$), and $$\mathbf{a} = (a_1,\ldots,a_n)$$ a point. Then, the partial derivative of $$f$$ at $$\mathbf{a}$$ in the $$i$$th variable $$x_i$$ is:

$$\frac{\partial}{\partial x_i}f(\mathbf{a}) = \lim_{h \to 0}\frac{f(a_1,\ldots,a_{i-1},a_i+h,a_{i+1},\ldots,a_n) - f(a_1,\ldots,a_{i-1},a_i,a_{i+1},\ldots,a_n)}{f(h)},$$

and the gradient of $$f$$ at $$\mathbf{a}$$ is:

$$\nabla f = \begin{bmatrix} \frac{\partial}{\partial x_1} f(\mathbf{a}) \\ \ldots \\ \frac{\partial}{\partial x_n} f(\mathbf{a}) \end{bmatrix}.$$

Fundamentally, backpropagation is a technique for computing partial derivatives quickly. For example, when training neural networks, we think of the cost as a loss function $$\ell$$ of the parameters $$W$$ of the network (i.e., the numbers describing how the network behaves). Therefore, the goal is to calculate derivatives of the cost concerning all the network parameters.