2
$\begingroup$

In this model from MusicNet, they set the initial weights of their neural network to all zeros.

self.linear = torch.nn.Linear(regions*k, m, bias=False).cuda()
torch.nn.init.constant(self.linear.weight, 0)

However, people normally randomize the initial weights as discussed in the following threads.

  1. Initialize perceptron weights with zero
  2. What are the cases where it is fine to initialize all weights to zero
  3. Danger of setting all initial weights to zero in Backpropagation

Basically, it means that we shouldn't set the initial weights to all zeros.

However, when I tried using random weights in the MusicNet model by commenting one line out

self.linear = torch.nn.Linear(regions*k, m, bias=False).cuda()
# torch.nn.init.constant(self.linear.weight, 0

The model doesn't learn anymore, the accuracy is very low (0.02 compared with 0.6 in the original model)

What I discovered

If the initial weights are random, the network seems unable to learn any meaningful weights. But when we set the initial weights to zero, the network can learn something meaningful.

Why is it so? I don't understand why only zero weights works in this model. enter image description here

$\endgroup$

1 Answer 1

0
$\begingroup$

Let's think of

learn something meaningful

as a minimum on the manifold along which the optimizer moves the state of the system until the state descents into the same minimum.

If you do a random initialization you throw the system into a random state on the manifold. But what if you had knowledge of a particular part of the manifold from which you know a direction to push towards (loss) to descent into that minimum. I believe that in your case the zero state of the weights is some fixpoint in the output space from which the optimization is eased with your choice of loss.

Theoretically, if you could define a path for the optimizer through the loss function, you could reach it as well, but it unlikely that you can accurately express it.

But that's just like, my opinion, man.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.