Why does this paper say that 0-1 loss is insensitive to scaling of weights in a neural network?

Question

When discussing capacity control using norms of weights in a neural network,this paper says the following(see P4):

Capacity control in terms of norm, when using a zero/one loss (i.e. counting errors) requires us in addition to account for scaling of the output of the neural networks, as the loss is insensitive to this scaling but the norm only makes sense in the context of such scaling. For example, dividing all the weights by the same number will scale down the output of the network but does not change the $0 / 1$ loss, and hence it is possible to get a network with arbitrary small norm and the same $0 / 1$ loss. Using a scale sensitive losses, such as the cross entropy loss, does address this issue (if the outputs are scaled down toward zero, the loss becomes trivially bad), and one can obtain generalization guarantees in terms of norm and the cross entropy loss.

I'm a little bit confused about this paragraph, since it seems that the 0-1 loss is not insensitive to weight scaling. For example, if we scale down the weights, the output would also be scaled down, so the 0-1 loss would definitely be changed(since the true labels are not changed).Can anyone explain to me which part of my understanding goes wrong? Also, I would like to know how the cross-entropy loss settles this problem.

Answer 1

Zero-one loss only really makes sense for (hard) classification, so the output of the network needs to already include a discretization. Presumably the authors have a particular system in mind, and scaling weights doesn't affect the discretized predictions.

Two possibilities (though very closely related) come to mind, though I can't tell whether the authors had either of them in mind. If the final output of the model is on a linear scale, and the decision is based on the sign of the output, then scaling weights will not change the sign and hence not the decision. Similarly, if the final output of the model is on a probability scale via a sigmoid activation, then scaling the weights will not change the log-odds's sign, hence will preserve whether the output is above or below 0.5, hence not change the final decision (assuming a 0.5 cutoff).