Backpropagation - simplest explanation

Question

Could you please explain, in simplest way, the algorithm (mathematical equation) of back-prop?

I have read lot of articles on it so I know what it is and understand the intuition behind it, but I still do not understand the equation of "upgrading/changing" neurons' attributes in layers.

Answer 1

You can refer to this answer here for a very detailed explanation and derivation of the backpropagation algorithm.

In general, backpropagation is based on the idea that it is possible to attribute an amount of blame to each parameter in your model for the resulting error. So this is what we do, first we will set all the parameters in our neural network (weights and biases) randomly. Then we will pass examples through the model, and at the output we will compute some loss function. Then using backpropagation we will tune our model parameters proportionally to their contribution to the incurred error.

Here are some other related answers that explain the complexities of backpropagation:

• Derivative of the sigmoid

• Quick run through of the equations dictating backpropagation

• Gradient descent, backpropagation and the partial derivative

• Computing the backpropagation

• Backpropagation in CNN

• Backpropagation with different activation functions

Answer 2

Note

This is my understanding. Please correct if any.

Gradient Descent

Backpropagation is to reduce the cost $J$ of the entire neural network (NN) and it is a problem to optimize the weight parameter $W$ to minimize the cost. Providing the cost function $J=f(W)$ is convex, the gradient descent $ W = W - \alpha f'(W)$ will result in the $Wmin$ which minimizes $J$. The hyperparameter $\alpha$ is called learning rate which we need to optimize too, but not in this answer.

$Y$ should be read as $J$ in the diagram. Imagine you are on the surface of a place whose shape is defined as $J=f(W)$ and you need to reach the point $Wmin$. There is no gravity so you do not know which way is toward the bottom but you know the function and your coordinate. How do you know which way you should go? You can find the direction from the derivative $f'(W)$ and move to a new coordinate by $ W = W - \alpha f'(W)$. By repeating this, you can get closer and closer to the point $Wmin$.

Chain Rule

NN is a chain of functions and the actions going on at the a neuron at the last layer can be described as below. In a real NN, sigmoid would not be used at the output nor as an activation function and the loss function gets multiple outputs from the last layer. However here, it is to illustrate the chain rule.

1. $X$ from a previous neuron gets into the product function $Y = g(W)$
2. $Y$ gets into the activation function $Z = h(Y)$ (using sigmoid)
3. $Z$ as an output gets into the loss function $J = L(Z)$ (using cross entropy log loss)

The cost function $J = f(W) = L( h(g(W)) )$ which is a chain of functions. $X$ is constant during the gradient calculation (until updated in the next forward cycle).

Gradient descent optimizes the parameter $W$ to minimize the cost $J$ providing $f$ is convex. We know the derivative $f'(W)$ via the chain rule. We only need to prove $f$ is convex. The actual cost $J$ is the sum from the all outputs, and the derivative of sum(+) is 1, hence omitted.

Backpropagation to the previous layer

The weight parameter of a neuron in the previous layer needs to be optimized with the gradient descent, too. The derivative $\frac{\Delta J}{\Delta X} = W*(Z-t)$ from the posterior layer needs to be incorporated in the chain rule in the neuron. This cascading incorporation of the derivative from posterior layer keeps going down to the downstream layers.

Example

Backpropagation from the output layer 2 to hidden layer 1 for the NN in the diagram.

The original code is from Coursera Week 4 Multi-class Classification and Neural Networks in the Coursera Machine Learning.

%------------------------------------------------------------------------
% Calculate the gradients at neurons in the layer 2 (output) and layer 1 (hidden).
% Not doing the gradient descent here.  
% m is the number of training images
%------------------------------------------------------------------------
for i = 1:m     
    %------------------------------------------------------------------------
    % i is training set index of X(including bias). X(i, :) is 401 data.
    % y is label
    %------------------------------------------------------------------------
    xi = X(i, :);
    yi = Y(i, :);
    
    % hi is the i th output of the hidden layer. H(i, :) is 26 data.
    hi = H(i, :);
    
    % oi is the i th output layer. O(i, :) is 10 data.
    oi = O(i, :);
    
    %------------------------------------------------------------------------
    % Calculate the gradients of Weight2 at layer 2 (output)
    %------------------------------------------------------------------------
    chained_derivative_for_weight2 = oi - yi;   % Read as (Z-t) 
    Weight2_grad = Weight2_grad + bsxfun(@times, hi, transpose(chained_derivative_for_weight2));
 
    %------------------------------------------------------------------------
    % Calculate the gradients of Weight1 at layer 1 (hidden)
    %------------------------------------------------------------------------
    derivative_from_posterior = sum(bsxfun(@times, Weight2, transpose(chained_derivative_for_weight2)));

    % Derivative of g(z): g'(z)=g(z)(1-g(z)) where g(z) is sigmoid(H_NET) in the neuron.
    dgz = (hi .* (1 - hi));

    % Incorporate the derivative from the posterior layer in the chain rule in the neuron
    chained_derivative_for_weight1 = dgz .* derivative_from_posterior
    
    % There is no input into H0, hence there is no weight for H0. Remove H0.
    chained_derivative_for_weight1 = chained_derivative_for_weight1(2:end);
    Weight1_grad = Weight1_grad + bsxfun(@times, xi, transpose(chained_derivative_for_weight1));
end

References

The original figure is from the Coursera. The reason using sigmoid and cross entropy log loss is because their derivatives result in such a simple formula $Z-t$, which is $(a - y)$ in the figure.

• How flexible is the link between objective function and output layer activation function?

Matrix

To handle the mini batch, need to handle the back propagation as Jacobian products. SeeProduct of Jacobians and chain Rule. Mathematics for Machine Learning 5.4 Gradients of Matrices has further explanations.

dL/dW.T

To calculate the impact of the changes in W.T on the total loss L of the neural network. Using the row-order matrix in the diagram.

The deduction will result in a different formula depending on what order of the matrix to use in X, W, and Jacobian, and how to organize weights in W. Andrew Ng used row-order and per-node weight grouping in Theta or W in his ML course at Coursera. CS231 of Stanford University by Justin Johnson used different weight grouping in W.