After reading the famous paper, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, I understand two things:-

  • He initilization borrows on the benefits of Xavier initialization except that the latter expected a linear activation and the prior accounts for ReLU non-linear activation. Infact they differ just by a factor of sqrt(2).

  • All the fuss is about layers having 0 mean and 1 std achieved by Xavier initialization but when ReLU is used the negative is clipped and so mean rises and thus the He initialzation introduces the sqrt(2) correction term which helps train a lot better.

So my question is as about the basic ResBlock which almost uninamously implemented as follows,

class ResBlock(nn.Module):

    def __init__(self,in_c):
        super(ResBlock, self).__init__()

        self.conv1 = nn.Conv2d(in_c,in_c, kernel_size=3, stride=1, padding=1, bias=True)
        self.conv2 = nn.Conv2d(in_c,in_c, kernel_size=3, stride=1, padding=1, bias=True)
        self.relu = nn.LeakyReLU(negative_slope=0.2, inplace=True)

    def forward(self, x):

        identity = x

        out = self.relu(self.conv1(x))
        #out = F.leaky_relu_(out, negative_slope=0.2)
        out = self.conv2(out)
        out = out + identity

        return out

Note that conv2 is not followed by ReLU and only Conv1 is followed by ReLU. Thus conv2 should have Xavier init and Conv1 have He init. But the dominant practice is to use He init for both. Isn't this wrong?


Alternatively, we can have conv2 have He initialization with 'leak parameter, a=1' and Conv1 have He initialization with 'leak parameter a=0'. The leak parameter is used to calculate the appropriate gain. But the dominant practice is to use He init for both with leak parameter set to 0. Isn't this wrong?

The 'leak parameter a' is the negative slope of the ReLU activation used after the conv layer. Check this for more information.




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.