My deep learning lecturer told us that if a hidden node has identical input weights to another, then the weights will remain the same over the training/there will be no separation. This is confusing to me, because my expectation was that you should only need the input or output weights to be different. Even if the input nodes are identical, the output weights would affect the backpropagated gradient at the hidden nodes and create divergence. Why isn't this the case?


It is fine if a hidden node has identical initial weights with nodes in a different layer, which is what I assume you mean by output weights. The problem with weight-symmetry arises when nodes within the same layer that are connected to the same inputs with the same activation function are initialized identically.

To see this, the output of a node $i$ within a hidden layer is given by $$\alpha_i = \sigma(W_i^{T}x + b) $$ where $\sigma$ is the activation function, $W$ is the weight matrix, $x$ is input, $b$ is bias.

If the weights $W_{i}=W_{j}$ are identical for nodes $i,j$ (note that bias is typically initialized to 0), then $\alpha_i = \alpha_j$ and the backpropagation pass will update both nodes identically.

  • $\begingroup$ "And the backpropagation pass will update both nodes identically" - this is what is confusing me? Doesn't the partial derivative depend on the nodes in the subsequent layers? And isn't the update amount simply a fraction times the partial derivative, so won't it change if the partial derivative changes? $\endgroup$
    – Casebash
    Jun 15 '20 at 12:04
  • $\begingroup$ Yes your intuition is correct, and in theory we can avoid this problem by updating each weight individually by computing the gradients via $\frac{\partial C}{\partial W_i}$, thus avoiding the symmetry problem. However, this is typically not practical since it is extremely computationally expensive. The backpropagation algorithm instead takes the weighted inputs of all nodes within a layer, computes its gradient, then does this recursively layer-by-layer. Thus hidden nodes with the same weights are multiplied by the same fraction (the gradient descent update). $\endgroup$
    – Adam
    Jun 15 '20 at 22:45
  • 1
    $\begingroup$ To clarify, the weight symmetry problem is not a mathematical problem with the theory of neural networks but arises due to the way gradients are computed via standard backpropagation algorithms. There are other ways to update the weights, such as asynchronous training via synthetic local gradients (Jaderberg et al. 2016). $\endgroup$
    – Adam
    Jun 15 '20 at 22:59
  • $\begingroup$ " The backpropagation algorithm instead takes the weighted inputs of all nodes within a layer" - Do you know where I can read more about this optimisation? $\endgroup$
    – Casebash
    Jun 16 '20 at 10:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.