In the $\delta$-rule which is used for error back propagation in neural networks, there is a factor $f'\!\left(\sum_i w_{ij}y_i\right)$, often written as $f'(\text{net}_i)$, which is just the derivative of the activation or transfer function evaluated at the weighted sum of the input. The derivative of $\tanh$ is bell-shaped so that values $\left|x\right|\geqslant3$ it is almost zero:


This means that units with a strong activation in either way have a more limited scope of action, where as values close to zero have more leeway. If I'm understanding it correctly, it keeps neurons that have already strongly decided one way or the other in their configuration. What is the purpose of this for weight updating? How would the networks behave differently if one would remove or change this factor?

As a reminder, the full $\delta$-rule looks like this for hidden neurons and output neurons:

$$\Delta w_{ij} = \eta\;\delta_j\;y_i$$

$$\delta_j = (\hat y_j - y_j)\;f'(\text{net}_j) \quad \text{or} \quad \delta_j = \sum\limits_k^K\left(w_{jk}\delta_k\right)\;f'(\text{net}_j)$$

where $\Delta w_{ij}$ is the change of the weight that connects the neuron $i^{\text{th}}$ neuron in the preceding layer with the $j^{\text{th}}$ neuron in the layer we are currently considering; $\eta$ is the learning rate, $y_j$ is the output of the current neuron, $\hat y_j$ is the teacher value for the $j^{\text{th}}$ output, $y_i$ is the output of the $i^{\text{th}}$, and $k$ is an index in the subsequent layer.


1 Answer 1


The equation that you show calculates (the negative of) a gradient to the objective function. The value $\delta_j$ is the rate of change of the objective function for an isolated change of output of a neuron before applying the activation function for the neuron. The factor of $f′(net_i)$ is required because it is correct - it is due to the chain rule applied twice - once due to the linear relationship linking weights between layers, and again to get the gradient pre-activation function in the next layer down.

An interpretation of $\delta_i$ is "the (negative) gradient of objective function with respect to a pre-activation value of a neuron the network". If, somehow, the pre-activation value could be changed - ignoring how it could be changed - then $\delta_i$ tells you the linear scale of the impact on the objective function, at least for very small changes.

This value is not normally used directly in weight updates, but is fed deeper backwards into the networks layers if there are yet lower layers to update, because it can be used for weight updates in those layers using the same rule.

The property of low gradients associated with strong activations is because you previously chose tanh as an activation function, and is not because of the way you calculate the gradient. Once you chose tanh, you were not really free to choose a different backpropagation gradient calculation. Or rather, there is not much to gain - and potentially a lot to lose - by doing so.

What is the purpose of this for weight updating?

The value shown is not used directly, but used to derive $\frac{\partial J}{\partial W_{ij}}$ gradients for weights in deeper layers (where $J$ is value of your objective function).

In terms of your equations

$$\frac{\partial J}{\partial W_{ij}} = -\delta_j y_i$$

This is very simple, so it has been folded in to your delta update rule. However, doing that is hiding something important. Out of all the intermediate values you calculate during back propagation, $\frac{\partial J}{\partial W_{ij}}$ is the key one. It is the gradient of the objective function with respect to parameters that you can change.

Once you have a set of gradients for $\frac{\partial J}{\partial W_{ij}}$, this can be used in various ways. Using this gradient directly for weight updates using the delta rule (as in your equation $\Delta W_{ij} = \eta \delta_j y_i$) is a basic approach. The purpose of this rule is to reduce the value of the objective function by taking a small step in the direction that you have just calculated will reduce the value of objective function. The step size needs to be small (multiplied by a factor $\eta$) because you have only measured the gradient at one point, and don't know for what step size the relationship still holds - complicated by the fact that you are making a similar step in multiple dimensions at once.

The delta update rule is a relatively weak update method compared to e.g. Nesterov momentum, Adagrad, Rmsprop optimisers which will use the calculated gradient as input, but adjust the update weights based on history of previous gradients. These optimisers can deal with low but consistent gradient values and make large updates.

How would the networks behave differently if one would remove or change this factor?

This is not advisable as effectively you would be optimising against a different network architecture than the feed-forward network you had constructed. The most likely way they would behave is to either diverge or settle on a larger value of your objective function than you would otherwise see. Making small changes (such as a minimum absolute gradient) can work, but usually such a tweak is best done consistently, by changing the activation function as well.

In a few edge cases, you might consider "lying" in the gradient calculation. A trivial case could be considered the ReLU activation function ($y = 0 (if x \lt 0), y = x (otherwise)$) where the gradient is technically undefined at $x=0$, but can safely be treated as either 0 or 1.

An alteration to updates that is more usually done (by e.g. momentum variants, Adagrad, RMSProp et al) is to use the gradient as-is, but then multiply update amounts by various factors based on history or estimates of higher-order derivatives, because you can guess that the gradient will not switch sign if you update a weight by a smaller or larger amount. These approaches replace the delta update rule with update mechanisms that are more robust when gradients become small.

  • $\begingroup$ I've added some information on the variable names I'm using to my question. $\endgroup$
    – Lenar Hoyt
    Commented Nov 12, 2015 at 16:09
  • $\begingroup$ OK, I think your $\delta_i$ etc are my $\frac{\partial J}{\partial z_i}$ - i.e. the gradients pre-activation-function. Whilst my answer assumes they are the matching $\frac{\partial J}{\partial a_j}$ . . . I will check and correct later. The main thrust of the answer is the same. $\endgroup$ Commented Nov 12, 2015 at 16:20
  • $\begingroup$ To be more precise, my $\delta_i$ are your $-\frac{\delta J}{\delta z_i}$. $\endgroup$
    – Lenar Hoyt
    Commented Nov 12, 2015 at 17:27
  • $\begingroup$ @mcb: OK, I think I am done updating the answer to better match your terms. $\endgroup$ Commented Nov 12, 2015 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.