1
$\begingroup$

I recently found this twitter thread from Andrej Karpathy. In it he states a few common mistakes during the development of a neural network.

  1. you didn't try to overfit a single batch first.
  2. you forgot to toggle train/eval mode for the net.
  3. you forgot to .zero_grad() (inpytorch) before .backward().
  4. you passed softmaxed outputs to a loss that expects raw logits.
  5. you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer. This one won't make you silently fail, but they are spurious parameters
  6. thinking view() and permute() are the same thing (& incorrectly using view)

I am specifically interested in an explanation or motivation for the fifth comment. Even more so given I have a network built akin to

self.conv0_e1 = nn.Conv2d(f_in, f_ot, kernel_size=3, stride=1, padding=1)
self.conv1_e1 = nn.Conv2d(f_ot, f_ot, kernel_size=3, stride=1, padding=1)
self.norm_e1 = nn.BatchNorm2d(num_features=f_ot, eps=0.001, momentum=0.01)
self.actv_e1 = nn.ReLU()
self.pool_e1 = nn.MaxPool2d(kernel_size=2, stride=2)

Where the torch.Conv2d has an implicit bias=True in the constructor.

How would I go about implementing the fifth point in the code sample above? Though, based on the second sentence of the point, it doesn't seem like this matters?..

$\endgroup$

1 Answer 1

1
$\begingroup$

Love this question. First, note the last thing said in the tweet you quote: "This one won't make you silently fail, but they are spurious parameters." Basically, this is a sort of mathematical quibble, but

To see what is happening here, consider what's happening in a BatchNorm2d layer vs Conv2d (quoting the PyTorch Docs):

BatchNorm2d: $y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta$

Conv2d: $\text{out}(N_i, C_{\text{out}_j}) = \text{bias}(C_{\text{out}_j}) + \sum_{k = 0}^{C_{\text{in}} - 1} \text{weight}(C_{\text{out}_j}, k) \star \text{input}(N_i, k)$

Both operations have an additive bias along the channel dimension. The two biases are either redundant or in conflict. If they're redundant, your model is performing extra, unnecessary computation. When they're in conflict, some parameters become useless, e.g. in the case where we set momentum=0 for BatchNorm2d, the preceding Conv2d layer will have a set of trainable parameters with no useful gradient.

You can fix this with Conv2d(..., bias=False, ...). Again, this is unlikely to have a significant impact on most networks, but it can be helpful and is good to know.

Follow-up edit:

The reason that this is relevant is that the BatchNorm2d layer is a linear operation. Multiple linear operations can always combined into a single one. So with Conv2d -> BatchNorm2d, the bias in Conv2d is redundant.

However, as long as you have activation functions, Conv2d layers are not linear, so with two conv layers, Conv2d -> Conv2d, you do not run into this problem.

So it's only the last layer before batchnorm that matters. (That said I've definitely seen examples where practitioners just turn off all biases for conv layers, and it doesn't seem to harm things.)

$\endgroup$
2
  • $\begingroup$ I need to think about this more to figure the proceeding question out for myself. But.. // Would the fix be implemented in every Conv2d layer or only Conv2d layers that proceed BatchNorm? $\endgroup$
    – KDecker
    Sep 30, 2021 at 16:58
  • $\begingroup$ Good point that I missed. The reason that this is relevant is that the BatchNorm2d layer is a linear operation. Multiple linear operations can always combined into a single one. So with Conv2d -> BatchNorm2d, the bias in Conv2d is redundant. However, as long as you have activation functions, Conv2d layers are not linear, so with Conv2d -> Conv2d, you do not run into this problem. So it's only the last layer that matters. (that said I've definitely seen examples where practitioners just turn off all biases for conv layers) $\endgroup$
    – Nix Searcy
    Sep 30, 2021 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.