4
$\begingroup$

I've taken a few online courses in machine learning, and in general, the advice has been to choose random weights for a neural network to ensure that your neurons don't all learn the same thing, breaking symmetry.

However, there were other cases where I saw people initializing using zero weights. Unfortunately, I can't remember what those were. I think it might have been non-neural-network cases, like a simple linear or logistic regression model (simple weights only on the inputs, leading directly to an output).

Are those cases safe for zero initialization? Alternatively, could we use random initialization in those cases too, just to stay consistent?

$\endgroup$
3
$\begingroup$

Whenever you have a convex cost function you are allowed to initialize your weights to zeros. The cost function of logistic regression and linear regression have convex cost function if you use MSE for, also RSS, linear regression and cross-entropy for logistic regression. The main idea is that for convex cost function you'll have just a single optimal point and it does not matter where you start, the starting point just changes the number of epochs to reach to that optimal point whilst for neural networks the cost function does not have just one optimal point. Take a look at here. About random initialization, you have to consider that you are not allowed to choose random weights which are too small or too big although the former was a more significant problem. If you choose random small weights you may have vanishing gradient problem which may lead to a network that does not learn. Consequently, you have to use standard initialization methods like He or Glorot, take a look at here and Understanding the difficulty of training deep feedforward neural networks.

Also, take a look at the following question.

$\endgroup$
  • $\begingroup$ But how do we get from "the neural network does not have only one optimal point" to "zero initialization is not OK"? $\endgroup$ – Stephen Apr 29 '18 at 20:50
  • $\begingroup$ @Stephen Initializing weights to zeros is not fine for deep networks due to symmetry problem. But for a single neuron it does not matter how the weights get initialized. I referred to cost function and number of optimal points to say why it matters or not for different situations. $\endgroup$ – Media May 1 '18 at 18:37
  • $\begingroup$ My point was just that whatever weights you choose, you will still have just a single point that you are starting from, so it would still be at risk of hitting whatever local minimum is closest nearby. I see how it helps us avoid the symmetry problem, I just don't see how it helps us avoid a local minimum. $\endgroup$ – Stephen May 3 '18 at 16:17
  • $\begingroup$ @Stephen basically in deep neural nets, we do not have local minimum, because its chance is low. Take a look at here. $\endgroup$ – Media May 3 '18 at 16:51
  • $\begingroup$ Oh, I thought that when you said "the cost function does not have just one optimal point" you meant that there are lots of local minima. $\endgroup$ – Stephen May 3 '18 at 17:00
1
$\begingroup$

Zeroing weights disables them. Yes, there are various applications of zero tensors (such as convex cost functions as you mention). Let's take the case of Neural Nets (NNs) and see if the math gives us more intuition:

$$ tensor\div0 = undefined\\ tensor * 0 = 0\\ tensor \cdot 0 = 0\\ $$

Example Graph #1: How would one disable a single synapse connected to the output layer?

                                                            

Math Example: Let $X$ be an input tensor of shape (1,2). Let $W$ be a weight tensor of shape (2,1). The dot product is represented here by the $\cdot$ symbol.

If all elements in tensor $W$ are zero:

$$ \begin{align*} X = \begin{matrix}[1&1]\\ \end{matrix}\ \ \ W = \begin{matrix}[0]\\ [0]\end{matrix}\ \end{align*} $$

$$ X \cdot W = [0]\\ $$

If all elements in tensor $W$ are randomly initialized (between -1 and 1):

$$ \begin{align*} X = \begin{matrix}[1&1]\\ \end{matrix}\ \ \ W = \begin{matrix}[0.24660266]\\ [0.05121049]\end{matrix}\ \end{align*} $$

$$ X \cdot W = [0.29781315]\\ $$

If one element in tensor $W$ is randomly set to zero:

$$ \begin{align*} X = \begin{matrix}[1&1]\\ \end{matrix}\ \ \ W = \begin{matrix}[0]\\ [0.05121049]\end{matrix}\ \end{align*} $$

$$ X \cdot W = [0.05121049]\\ $$

Aha, some intuition! Set an element of your weight vector $W$ to zero to disable it.

Example Graph #2: As complexity scales, so too does our potential loss of control over architecture.

When you need to tweak the details, it is important to have tools to change nodes and edges on a per unit basis. Zero weighting gives you that ability.

This idea generalizes to CNNs, GANs, RNNs, etc. Look at the particular algorithm and go layer by layer. What are the designers trying to accomplish?

$\endgroup$
  • $\begingroup$ My question was about initializing weights to zero, though, not about setting them to zero at some later stage (which I think would be dropout regularization). So I'm not sure how this answer really addresses my point, but maybe I'm missing something. $\endgroup$ – Stephen Apr 29 '18 at 20:45
  • $\begingroup$ Initializing all weights to zero would effectively disable that weight tensor. $tensor \cdot W = [0]$ Randomly initializing elements of the weight tensor to zero would disable those individual synapses. Is your question more about the application of a particular algorithm? I tried to provide a general mathematics example that you could apply broadly. $\endgroup$ – abrahamrhoffman Apr 30 '18 at 2:39
  • $\begingroup$ No, it's just about initializing regardless of what algorithm you're using. While it's technically correct that setting the weights to zero disables them, I think it misses the point because initialization is something that happens at the very beginning, and then we start training and updating the weights. So any disabling doesn't last for very long - it ends as soon as we update the weights for the first time. The issue is what initializations are best for that upcoming training process. $\endgroup$ – Stephen Apr 30 '18 at 14:06
  • $\begingroup$ I see what you're getting at. Initializing weights to zero, before the first back-prop pass sets higher priority on other layers that have not been zeroed. To talk more about a specific algorithm, and initializing a specific layer's weights to zero - do you have an example we could discuss? $\endgroup$ – abrahamrhoffman Apr 30 '18 at 22:19
  • $\begingroup$ The question was about initializing all to zero versus all to random, not only some. I suppose one specific example would be doing it with LogisticRegression. $\endgroup$ – Stephen May 1 '18 at 2:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.