Neural networks, optimization math intuition

Question

When I look into the following partial derivative, I see it as being the key element of any optimization algorithm out there. Correct me if I'm wrong, but this gets us the slope of the loss function, so we can go opposite to that slope, therefore minimizing the loss.

$$\frac{\partial \theta}{\partial \mathcal{L}}$$

where: $\theta$ is the weights, and the $\mathcal{L}$ is the loss;

---

Does that make sense? Is there any other calculation step that is arguably more fundamental to the optimization of neural networks other than this derivative?

This topic is specially important for me right now, because I was thinking of tattoing this derivative, as a cool A.I. tattoo, and I want it to be fundamental and simple.

Answer 1

Note that $\frac{\partial L}{\partial \theta}$ is different from $\frac{\partial \theta}{\partial L}$. What you tried to describe seems to be $\frac{\partial L}{\partial \theta}$ where $\theta$ is a variable. If $\theta$ is high dimensional, sometimes we just use the $\nabla$ notation.

Gradient descent is

$$\theta_{n+1}=\theta_n-\gamma \nabla L(\theta_n)$$

• Not everything is differentiable and gradient might not be well defined for some optimization problem.

• In the event that there are constraints, $L$ might need to take the role of Langragian rather than the objective function.

• Gradient descent is just a means to find the parameters for a model. Gradient based approach seems to be the norm for now but things can change.

What you proposed to tattoo is just "gradient" or "slope". Not objection but just want to let you know what you are doing.

Answer 2

Apart from your tattoo, in gradient descent, the loss function needs to be minimised which is our objective function in this case.

The Gradient Descent update rule states that,

$\Large \theta_{ij} = \theta_{ij} - \frac{\partial L}{\partial \theta_{ij}}$

Where $\theta$ is the parameter which needs to be optimised. This is the fundamental equation of gradient descent which is used as an optimization algorithm in nearly all AI/ML tasks.