In this model from MusicNet, they set the initial weights of their neural network to all zeros.
self.linear = torch.nn.Linear(regions*k, m, bias=False).cuda() torch.nn.init.constant(self.linear.weight, 0)
However, people normally randomize the initial weights as discussed in the following threads.
- Initialize perceptron weights with zero
- What are the cases where it is fine to initialize all weights to zero
- Danger of setting all initial weights to zero in Backpropagation
Basically, it means that we shouldn't set the initial weights to all zeros.
However, when I tried using random weights in the MusicNet model by commenting one line out
self.linear = torch.nn.Linear(regions*k, m, bias=False).cuda() # torch.nn.init.constant(self.linear.weight, 0
The model doesn't learn anymore, the accuracy is very low (0.02 compared with 0.6 in the original model)
What I discovered
If the initial weights are random, the network seems unable to learn any meaningful weights. But when we set the initial weights to zero, the network can learn something meaningful.