After reading the famous paper, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, I understand two things:-
He initilization borrows on the benefits of Xavier initialization except that the latter expected a linear activation and the prior accounts for ReLU non-linear activation. Infact they differ just by a factor of
sqrt(2)
.All the fuss is about layers having 0 mean and 1 std achieved by Xavier initialization but when ReLU is used the negative is clipped and so mean rises and thus the He initialzation introduces the sqrt(2) correction term which helps train a lot better.
So my question is as about the basic ResBlock which almost uninamously implemented as follows,
class ResBlock(nn.Module):
def __init__(self,in_c):
super(ResBlock, self).__init__()
self.conv1 = nn.Conv2d(in_c,in_c, kernel_size=3, stride=1, padding=1, bias=True)
self.conv2 = nn.Conv2d(in_c,in_c, kernel_size=3, stride=1, padding=1, bias=True)
self.relu = nn.LeakyReLU(negative_slope=0.2, inplace=True)
def forward(self, x):
identity = x
out = self.relu(self.conv1(x))
#out = F.leaky_relu_(out, negative_slope=0.2)
out = self.conv2(out)
out = out + identity
return out
Note that conv2
is not followed by ReLU and only Conv1
is followed by ReLU. Thus conv2
should have Xavier init and Conv1
have He init. But the dominant practice is to use He init for both. Isn't this wrong?
---EDIT---
Alternatively, we can have conv2
have He initialization with 'leak parameter, a=1' and Conv1
have He initialization with 'leak parameter a=0'. The leak parameter is used to calculate the appropriate gain. But the dominant practice is to use He init for both with leak parameter set to 0. Isn't this wrong?
The 'leak parameter a' is the negative slope of the ReLU activation used after the conv layer. Check this for more information.
Thankyou