When discussing capacity control using norms of weights in a neural network,this paper says the following(see P4):
Capacity control in terms of norm, when using a zero/one loss (i.e. counting errors) requires us in addition to account for scaling of the output of the neural networks, as the loss is insensitive to this scaling but the norm only makes sense in the context of such scaling. For example, dividing all the weights by the same number will scale down the output of the network but does not change the $0 / 1$ loss, and hence it is possible to get a network with arbitrary small norm and the same $0 / 1$ loss. Using a scale sensitive losses, such as the cross entropy loss, does address this issue (if the outputs are scaled down toward zero, the loss becomes trivially bad), and one can obtain generalization guarantees in terms of norm and the cross entropy loss.
I'm a little bit confused about this paragraph, since it seems that the 0-1 loss is not insensitive to weight scaling. For example, if we scale down the weights, the output would also be scaled down, so the 0-1 loss would definitely be changed(since the true labels are not changed).Can anyone explain to me which part of my understanding goes wrong? Also, I would like to know how the cross-entropy loss settles this problem.