Differentiable vs Non Differentiable loss function in ML

Question

ML Question: What is a differentiable loss function and why does it matter?

For example, for a given Input training set; the loss function is: $L(y,F(x))$

Is this differentiable? How? And why does it matter?

Answer 1

In Layman's terms:

1. A loss function may be differentiable or else, depending on its exact form.

2. We like differentiable function because it allows the use of gradient-based optimization methods e.g. SGD. On the other hand, optimization methods for non-differentiable functions e.g. genetic algorithm are usually harder/takes longer to compute.

3. While training a model, we search for the minimum of the loss function, during which the optimization method is computed for a lot of times; thus a differentiable loss function is preferred for efficiency.

Answer 2

A loss function is differentiable if you can compute its derivative with respect to the model parameters.

In your example, there is not enough information to say if the loss function is differentiable, because we don't know how $L$ and $F$ are defined.

An example of differentiable loss function is the mean squared error, that is $\mathbf{L} = \frac{1}{n}\sum_{i=1}^n(y_i - F(x_i))^2$, where $F$ is our model (e.g. a neural network), which must also be differentiable.

Differentiable loss functions matter for gradient-based optimization approaches, like gradient descent for neural networks, because to be able to apply them you need the loss function (and the model) to be differentiable.

Answer 3

A function $f: A \to \mathbb{R}$, with $A \subset \mathbb{R}$, where $\mathbb{R}$ is the set of real numbers, is differentiable at $a \in A$ if its derivative:

$$f'(a) = \lim_{h \to 0}\frac{f(a+h) - f(a)}{f(h)}$$

exists, which implies that $f$ is continuous at $a$. The function $f$ is said to be differentiable on $A$ if it is differentiable at every point of $A$.

Differentiable loss functions are the bread and butter in training neural networks.

Historically, artificial intelligence (AI), and thus machine learning, started in 1956 with the proposal of McCharty et al. for a two-month, 10-person summer conference at Dartmouth College. Without going too deep with the chronological milestones, it is important to stress that many promises by AI researchers of future successes failed. The latter started what is called an AI winter. The return of neural networks came in 1986 thanks to the influential work on the backpropagation algorithm by Rumelhart, Hinton, and Williams.

To understand backpropagation, you must comprehend what partial derivatives are and what the gradient of a function is. Let $A \subset \mathbb{R}^n$ be an open set on $\mathbb{R}^n$, $f: A \to \mathbb{R}$ a function of $n$ variables (i.e., $f(x_1,\ldots,x_n)$), and $\mathbf{a} = (a_1,\ldots,a_n)$ a point. Then, the partial derivative of $f$ at $\mathbf{a}$ in the $i$th variable $x_i$ is:

$$\frac{\partial}{\partial x_i}f(\mathbf{a}) = \lim_{h \to 0}\frac{f(a_1,\ldots,a_{i-1},a_i+h,a_{i+1},\ldots,a_n) - f(a_1,\ldots,a_{i-1},a_i,a_{i+1},\ldots,a_n)}{f(h)},$$

and the gradient of $f$ at $\mathbf{a}$ is:

$$ \nabla f = \begin{bmatrix} \frac{\partial}{\partial x_1} f(\mathbf{a}) \\ \ldots \\ \frac{\partial}{\partial x_n} f(\mathbf{a}) \end{bmatrix}. $$

Fundamentally, backpropagation is a technique for computing partial derivatives quickly. For example, when training neural networks, we think of the cost as a loss function $\ell$ of the parameters $W$ of the network (i.e., the numbers describing how the network behaves). Therefore, the goal is to calculate derivatives of the cost concerning all the network parameters.