1
$\begingroup$

In this model from MusicNet, they set the initial weights of their neural network to all zeros.

self.linear = torch.nn.Linear(regions*k, m, bias=False).cuda()
torch.nn.init.constant(self.linear.weight, 0)

However, people normally randomize the initial weights as discussed in the following threads.

  1. Initialize perceptron weights with zero
  2. What are the cases where it is fine to initialize all weights to zero
  3. Danger of setting all initial weights to zero in Backpropagation

Basically, it means that we shouldn't set the initial weights to all zeros.

However, when I tried using random weights in the MusicNet model by commenting one line out

self.linear = torch.nn.Linear(regions*k, m, bias=False).cuda()
# torch.nn.init.constant(self.linear.weight, 0

The model doesn't learn anymore, the accuracy is very low (0.02 compared with 0.6 in the original model)

What I discovered

If the initial weights are random, the network seems unable to learn any meaningful weights. But when we set the initial weights to zero, the network can learn something meaningful.

Why is it so? I don't understand why only zero weights works in this model. enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.