I found that there is no common resource and well defined definition for "Gradient norm", most search results are based on ML experts providing answers which involves gradient norm or papers which reference it and provide a single sentence intro to it.
Is there any well defined resource I can refer to get a concrete understanding of it ? Thank you
The concept of norm comes from functional analysis (and is found both in linear algebra and in optimization methods and other fields). There are several types of norm. The one in question is the norm of Euclidean space (which is the square root of the rocky product of vectors).
(a,a)^(1/2) = ||a|| (a,a) = a_1a_1+...+a_na_n when a = (a_1, ..., a_n)
The gradient is the point of the greatest growth of the function at the selected point (in fact, it is a vector of partial derivatives of the function from n variables).
] f(x,y) --> grad u = ( f′x(x,y), f′y(x,y))
So the gradient norm is the rock product of the vector of partial derivatives by the vector of partial derivatives.
||grad u|| = (grad u, grad u)^1/2 = ((f′x(x,y))^2 + f’y(x,y)^2)^1/2
This indicator is used in the gradient method of finding the absolute minimum/maximum of functions.
The method has a number of advantages (such as simplicity and stability) and at the same time a number of disadvantages (such as a degraded method with an increase in the dimensionality of the function or a sawtooth effect of finding a minimum / maximum when choosing the wrong step).
The L2 gradient norm is simply the sum of the squares of the individual gradients. The L1 norm would be the sum of absolute values of the gradients, though this tends to be less common imo. The basic idea is to scale/clip gradients to prevent vanishing/exploding gradients. This is explained in a number of articles online: https://machinelearningmastery.com/how-to-avoid-exploding-gradients-in-neural-networks-with-gradient-clipping/ hth.
