I want to introduce Batch Normalization in my C++/CUDNN implementation of CNN. The implementation is currently performing well (without BN) on the MNIST dataset.
I am using the CUDNN implementation of Batch Norm, but after having read the Batch Norm paper and the CUDNN documentation carefully, still there are some points that are not clear to me.
- From what I understood, batch norm layers are placed between the output of convolutional/dense layers and the non-linearity, like:
u --->[conv/dense (weights W)]--->[BN]--->[ReLU]---> y = ReLU(BN(W*u))
In some literature I read that placing the batch norm layer after the non-linearity is a better option. Is this true?
The CUDNN documentation says to use the BATCHNORM_MODE_SPATIAL for convolutional layers, and BATCHNORM_MODE_PER_ACTIVATION for dense layers. However, in another implementation (YOLO / Darknet), I only see BATCHNORM_MODE_SPATIAL being used. So I ask myself whether SPATIAL is also OK for dense layers?
As mentioned, from what I understood, BATCHNORM_MODE_SPATIAL should be used for conv layers: but on their input (i.e. before the convolution), or on their output (i.e. after the convolution) ?
From what I understood, it is to be used on the output, like:
u ---->[ conv (weights W) ] ---->[BN with mode SPATIAL]------>[ReLU]---> y
is this correct?
I see that in Darknet / YOLO they do not update the parameters explicitly with the deltas obtained from the cudnnBatchNormalizationBackward(). Does this method also perform the parameters update directly? I don't find any info on this, and I am actually applying a Gradient Descent update after having obtained the Gamma and Beta gradients from the cudnnBatchNormalizationBackward method.
I see that in the CUDNN implementation, a Forward method for Inference is available. Is it OK to completely skip the BN units during the inference phase? In this scenario, I would train with the BN units, and remove them from the network during inference.
I really thank you in advance for your patience and your kind help. I tried to get rid of these questions alone for quite long, but I am now blocked... your help is really appreciated.