In this paper (page 1 abstract) which considers regularization technique, the author used the word "norm" - what does it stand for? Is it related to Batch Normalization / L1 or L2 Normalization?

"We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing the squared distance between successive hidden states’ norms"

Googling "what is a norm Neural Net" doesn't provide any links

  • 2
    $\begingroup$ The state of a neural network is a vector or tensor, and these mathematical objects have norms. $\endgroup$
    – Emre
    Feb 27 '18 at 2:44

A norm is a concept in linear algebra which assigns a size to a vector. Many different norms exist you can read up on their many uses here.

In this paper, they are training a recurrent neural network. These are used for time series data. The assumption is a hidden state should be similar for successive time inputs $t$ and $t-1$. Thus, the author proposes an additional constraint based on the relative size of the successive hidden state vectors as

$||h_{t}||_2 - ||h_{t-1}||_2$.

The more different the norm of these successive states, the greater the cost will be.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.