In the context of machine learning, what is the relationship between "Normalization" and "norm"

Question

"Normalization" and "norm" are used a lot in machine learning

In statistics and applications of statistics, normalization can have a range of meanings.

In the simplest cases, normalization of ratings means adjusting values measured on different scales to a notionally common scale, often prior to averaging. In more complicated cases, normalization may refer to more sophisticated adjustments where the intention is to bring the entire probability distributions of adjusted values into alignment. In the case of normalization of scores in educational assessment, there may be an intention to align distributions to a normal distribution. A different approach to normalization of probability distributions is quantile normalization, where the quantiles of the different measures are brought into alignment.

In linear algebra, functional analysis, and related areas of mathematics, a norm

is a function that assigns a strictly positive length or size to each vector in a vector space—except for the zero vector, which is assigned a length of zero. A seminorm, on the other hand, is allowed to assign zero length to some non-zero vectors (in addition to the zero vector). A norm must also satisfy certain properties pertaining to scalability and additivity which are given in the formal definition below.

in the context of machine learning, what is the relationship between "Normalization" and "norm"

Answer 1

In the context of deep learning, normalization usually refers to the process of subtracting the mean and dividing by the standard deviation:

$$ \hat{x_i}=\frac{x_i - \mu}{\sigma} $$

This kind of normalization is not related with the norm of a vector. Instead, it refers to the statistical notion you referred aimed at rescaling the values. In a statistical context, this approach is sometimes referred to as "standardization" of a random variable, which makes its mean $0$ and its standard deviation $1$, assuming the original variable follows a normal distribution.

---

There exist multiple types of normalization in deep learning, depending on what we normalize over. The most used ones are instance normalization, batch normalization and layer normalization. These are explained graphically in the image below, which is borrowed from article Group Normalization:

Each subplot shows a feature map tensor, with N as the batch axis, C as the channel axis, and (H, W) as the spatial axes. The pixels in blue are normalized by the same mean and variance, computed by aggregating the values of these pixels.

Each type of normalization has a different purpose, and some of them have shown their practical success but it's not clear why they work well. These are some authoritative references for the mainstream normalization strategies:

• Batch normalization: [Ioffe and Szegedy, 2015], [Santurkar et al., 2018]
• Instance normalization [Ulyanov et al., 2016]
• Layer normalization: [Ba et al., 2016]

Note that some of the normalization variants need further processing apart from the value normalization itself. For instance, with batch normalization you need to store $\mu$ and $\sigma$ to use them at inference time.

Answer 2

Let's take L2 norm as an example.

This figure from wiki illustrates "in a right-angled triangle the square of the hypotenuse is equal [to the sum of] the squares of the two other sides"

given a = 3, b = 4, then c = 5.

consider c as a vector,
$ c = \begin{bmatrix} 3 \\ 4 \\ \end{bmatrix} $

the L2 norm of c is the length (magnitude) of c = 5.

to normalize the vector c, divide each component by its norm 5, then the normalized vector = $ \begin{bmatrix} 3/5 \\ 4/5 \\ \end{bmatrix} $

this processing is called normalization.