Shouldn't L2 regularization be normalized for the number of nodes in a layer?

Question

I'm naively thinking it would be better to normalize L2 regularization to the number of elements in a tensor, but I don't see anyone doing that. What am I missing?

I'd say in a multi layer fully connected NN:

x  = input            # vector length n_in
w1 = weights          # shape n_in x n_1
b_1 = bias            # shape n_1
h_1 = Relu(w1 * x + b_1)

w2 = weights          # shape n_1 x n_2
b_2 = bias            # shape n_2
h_2 = Relu(w2 * h_1 + b_2)

Then the L2 loss is normally defined as:

L2 = lambda * ( sum((w1) ^2) + sum((w2) ^2) )

If the first layer is much bigger than the second layer, than the first term in above L2 definition is much bigger, effectively making the second term useless. I'm thinking this would be better:

L2 = lambda * ( sum((w1) ^2)/(n_in*n_1) + sum((w2) ^2)/(n_1*n_2) )

This would also make the hyper parameter lambda independent of the number of nodes you choose. However, I don't see anybody doing this so I suspect I'm missing something. What is it?

Answer 1

However, I don't see anybody doing this so I suspect I'm missing something. What is it?

You are concentrating on how this extra loss term affects the absolute value of the loss function. This is not really relevant. Many forms of regularisation are not expressed as changes to loss function at all - framing L2 regularisation as part of the loss function is convenient because it allows us to re-use the existing weight adjustment logic. However, functionally it is almost identical to a weight decay term applied per batch independently of loss function, e.g. multiply all weights by 0.99 every batch.

If you use L2 regularisation loss, then you will almost certainly want to track a separate metric such as mean square error (i.e. your original loss function before considering regularisation), when comparing test results between different levels of regularisation.

That said, there is no reason that you cannot have a different L2 weight param per layer, and many frameworks will support this. Sometimes this could result in better generalisation. It is not done much in practice because it adds yet more dimensions to search when optimising hyper-parameters to a problem.

There is no reason expect better performance by aiming for roughly equal loss values from regularisation per layer. However, if that is your goal, then a naive scaling by inverse layer size $\frac{1}{N_{in} \times N_{out}}$ would not do it. That is because the weights in a trained network will already tend to scale due to the training targets so that the mean squared value is proportional to $\frac{1}{N_{in} + N_{out}}$. So your actual scaling factor to make the L2 regularisation loss per layer roughly the same would be $\frac{N_{in} + N_{out}}{N_{in} \times N_{out}}$. In code this might look like:

L2 = lambda * (
  sum((w1) ^2)*(n_in+n_1)/(n_in*n_1) + sum((w2) ^2)*(n_1+n_2)/(n_1*n_2)
)

(and of course matching changes to loss gradients). Although I will stress again that I do not expect this change to result in better generalisation.