# Why don't different output weights break symmetry?

My deep learning lecturer told us that if a hidden node has identical input weights to another, then the weights will remain the same over the training/there will be no separation. This is confusing to me, because my expectation was that you should only need the input or output weights to be different. Even if the input nodes are identical, the output weights would affect the backpropagated gradient at the hidden nodes and create divergence. Why isn't this the case?

It is fine if a hidden node has identical initial weights with nodes in a different layer, which is what I assume you mean by output weights. The problem with weight-symmetry arises when nodes within the same layer that are connected to the same inputs with the same activation function are initialized identically.

To see this, the output of a node $$i$$ within a hidden layer is given by $$\alpha_i = \sigma(W_i^{T}x + b)$$ where $$\sigma$$ is the activation function, $$W$$ is the weight matrix, $$x$$ is input, $$b$$ is bias.

If the weights $$W_{i}=W_{j}$$ are identical for nodes $$i,j$$ (note that bias is typically initialized to 0), then $$\alpha_i = \alpha_j$$ and the backpropagation pass will update both nodes identically.

• "And the backpropagation pass will update both nodes identically" - this is what is confusing me? Doesn't the partial derivative depend on the nodes in the subsequent layers? And isn't the update amount simply a fraction times the partial derivative, so won't it change if the partial derivative changes? Jun 15 '20 at 12:04
• Yes your intuition is correct, and in theory we can avoid this problem by updating each weight individually by computing the gradients via $\frac{\partial C}{\partial W_i}$, thus avoiding the symmetry problem. However, this is typically not practical since it is extremely computationally expensive. The backpropagation algorithm instead takes the weighted inputs of all nodes within a layer, computes its gradient, then does this recursively layer-by-layer. Thus hidden nodes with the same weights are multiplied by the same fraction (the gradient descent update).