4
$\begingroup$

As mentioned in the question, I have some issues understanding what are the differences between those terms.

From what I have understood:

  1. Forward pass: compute the output of the network given the input data

  2. Backward pass: compute the output error with respect to the expected output and then go backward into the network and update the weights using gradient descent ecc...

What is backpropagation then? Is it the combination of the previous 2 steps? Or is it the particular method we use to compute dE/dw? (chain rule ecc...)

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

In a narrow sense backpropagation only refers to the calculation of the gradients. So it does, for example, not include the update of any weights. But usually it is used refering to the whole backward pass.

Also see Wikipedia.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Exactly, the weights (from the mathematical point of view) are actually updated using gradient descent, which needs dE/dw, which is calculated with backpropagation, right? $\endgroup$
    – Mattia Surricchio
    Commented Jan 13, 2020 at 16:19
  • 1
    $\begingroup$ Yes. Moreover, you could say backpropagation refers primarily to feedforward networks while, for example, RNNs use backpropagation through time. $\endgroup$
    – Jonathan
    Commented Jan 13, 2020 at 16:27
1
$\begingroup$

In brief, backpropagation references the idea of using the difference between prediction and actual values to fit the hyperparameters of the method used. But, for applying it, previous forward proagation is always required. So, we could say that backpropagation method applies forward and backward passes, sequentially and repeteadly.

Your machine learning model starts with random hyperparameter values and makes a prediction with them (forward propagation). Then it compares with real values while adjusting those random initial values (backpropagation), trying to minimize the error (depending of your objective function and optimization method applied). And then, it starts over again, until you reach the stopping criteria.

You may find a better explanation in this question.

Furthermore, you will find more topic explanation here.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So backward pass and backpropagation are the same thing? I saw many books/articles on medium that used these two terms, sometimes referring only to the backward part, sometimes referring to the whole process of forward + backward $\endgroup$
    – Mattia Surricchio
    Commented Jan 13, 2020 at 16:18
  • 1
    $\begingroup$ As I said, you can't go back without going straight first. But sounds good for me the concept of using forward/backward pass for specifying JUST the step of going forward or backward while backpropagation includes both. However, this is a lenguage matter. Under my point of view, going backward always include going forward first, so, it's a concept elided. $\endgroup$
    – Dave
    Commented Jan 13, 2020 at 16:23
0
$\begingroup$

Forward Pass is same as Forward Propagation and Backward Pass is same as Backpropagation.

Forward Pass(Forward Propagation) is to calculate a model's predictions with true values(train data), working from input layer to output layer.

Backward Pass(Backpropagation) is to calculate a gradient using the mean(average) of the sum of the losses(differences) between the model's predictions and true values(train data), working from output layer to input layer.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.