Background

cs231n has the question regarding how to initialize weights.

Question

Please confirm or correct my understandings. I think the weight value will be the same in all the neurons with ReLU activation.

When W = 0 or less in all neurons, the gradient update on W is 0. So W will stay 0. When W = 1 (or any constant), the gradient update on W is the same value in all neurons. So W will be changing but the same in all neurons.

When the weight is the same, o/p will be the same for all the Neurons in every Layer.

$$\hspace{3cm}$$

Hence, during backpropagation, all the Neurons in a particular layer will get the same Gradient portion. So, the weight will change by the same amount.

But, the very first layer (connected to input Features) will work normally as its input is the actual Features itself which is different always.

I'll make this as explicit and non handwavy as possible, at the risk of being boring.

Let us take a small toy example: a network with two inputs $$x_1$$ and $$x_2$$, two hidden layers with two units each and one output. For simplicity, we will assume that the activation function is the identity function (i.e. no activation) and that the objective function is the output itself (not very realistic, but for our purposes this could mean our cost function is meant to drive the output to the zero function).

Below are the outputs of each unit expressed as a function of their inputs.

$$y = w_9y_3 + w_{10}y_4$$ $$y_3 = w_5y_1 + w_7y_2$$ $$y_4 = w_6y_1 + w_8y_2$$ $$y_1 = w_1x_1 + w_3x_2$$ $$y_2 = w_2x_1 + w_4x_2$$

Take a look at $$w_9$$: $$\frac{\partial y}{\partial w_9} = y_3 = w_5y_1 + w_7y_2 = w_5(w_1x_1 + w_3x_2) + w_7(w_2x_1 + w_4x_2)$$. The update of $$w_9$$ depends on $$w_1$$, $$w_2$$, $$w_3$$, $$w_4$$, $$w_5$$ and $$w_7$$: if we consider the diagram above to be a directed acyclic graph, the weights of all the edges on paths leading to $$y_3$$.

In our toy example, updates are simple: $$w_{i, n+1} = w_{i, n} - \eta \frac{\partial y}{\partial w_i}(w_{i, n})$$, where $$i$$ denotes the weight in the diagram and $$n$$ denotes the iteration.

If all weights are initialized to a constant $$a$$, this amounts to $$w_9 = a$$ and $$\frac{\partial y}{\partial w_9} = a(ax_1 + ax_2) + a(ax_1 + ax_2)$$.

Let us also take a look at $$w_{10}$$: $$\frac{\partial y}{\partial w_{10}} = w_6(w_1x_1 + w_3x_2) + w_8(w_2x_1 + w_4x_2)$$. In this case, $$w_{10} = a$$ and $$\frac{\partial y}{\partial w_{10}} = a(ax_1 + ax_2) + a(ax_1 + ax_2)$$.

As can be seen, the initial value of both $$w_9$$ and $$w_{10}$$ is the same and their updates are the same too. On the first iteration, they will have changed after the update, but they will still be equal to each other. Not necessarily a bad thing, but if they're equal, you might just as well keep only one of them in memory. If you're keeping both, they should be different.

What happens on the next iteration? Looking at the derivatives with respect to $$w_9$$ and $$w_{10}$$ previously derived, we notice that if $$w_5 \neq w_6$$ or $$w_7 \neq w_8$$ the two updates will be different, so let's take a look at those then!

$$\frac{\partial y}{\partial w_5} = \frac{\partial y}{\partial y_3} \frac{\partial y_3}{\partial w_5} + \frac{\partial y}{\partial y_4} \frac{\partial y_4}{\partial w_5} = w_9y_1 + w_{10} \cdot 0 = w_9(w_1x_1 + w_3x_2)$$

$$\frac{\partial y}{\partial w_6} = w_{10}y_1 = w_{10}(w_1x_1 + w_3x_2)$$

$$\frac{\partial y}{\partial w_7} = w_9(w_2x_1 + w_4x_2)$$

$$\frac{\partial y}{\partial w_8} = w_{10}(w_2x_1 + w_4x_2)$$

We can see that for the same reasons as before, after the first iteration, $$w_5$$ and $$w_6$$ will be the same, just like $$w_7$$ and $$w_8$$ will be the same. This, combined with the fact that $$w_9$$ and $$w_{10}$$ were equal after the first iteration, means that $$w_9$$ and $$w_{10}$$ will still be equal after the second iteration.

We can already say that $$w_5 = w_6$$, $$w_7 = w_8$$ and $$w_9 = w_{10}$$ ad infinitum.

Finally, let us take a look at the weights of the first hidden layer, $$w_1$$, $$w_2$$, $$w_3$$ and $$w_4$$.

$$\frac{\partial y}{\partial w_1} = (w_9w_5 + w_{10}w_6)x_1$$

$$\frac{\partial y}{\partial w_2} = (w_9w_7 + w_{10}w_8)x_1$$

$$\frac{\partial y}{\partial w_3} = (w_9w_5 + w_{10}w_6)x_2$$

$$\frac{\partial y}{\partial w_4} = (w_9w_7 + w_{10}w_8)x_2$$

Here, $$x_1$$ and $$x_2$$ don't have to be equal, so $$w_1$$ and $$w_2$$ can be different from $$w_3$$ and $$w_4$$. However, if we initialize all weights to $$a$$, $$w_1 = w_2$$ and $$w_3 = w_4$$ after the first iteration, which means that $$w_5 = w_6 = w_7 = w_8$$ after the second iteration (not just the first).

To conclude, we can say that $$w_1 = w_2$$, $$w_3 = w_4$$, $$w_5 = w_6 = w_7 = w_8$$ and $$w_9 = w_{10}$$ ad infinitum!

Generalizing this toy example, we can state that if we initialize all weights to the same value $$a$$, the weights of a layer will all be equal forever, except for the first layer, which keeps a semblance of normalcy. In other words, using only one weight per layer is enough.

A special case of initializing all weights to the same value is initializing them all to 0. If we look once more at the partial derivatives (which we use to update our weights), we can see that they will all be zero, always! In this case, the weights never get updated at all, so our model becomes entirely useless.

A very pretty visualization for the effects of different basic weight initialization techniques can be viewed here.