Why is backpropagation used for finding the loss gradient?

Question

I am relatively new to the world of machine learning. After getting a general idea of the concept, I tried creating a program for training a deep learning network from scratch. My goal was to use as little outside resources/help as possible and figure out every step and issue on my own. Now that I'm done, I'm comparing the differences between my approach and the solutions that were created by people who know what they're doing. One of these differences I've seen is the backpropagation algorithm, and I'm in the process of trying understanding it (but am struggling a bit).

While researching the algorithm, most sites describe it as a method of calculating the loss gradient of the network. However, the method that I used for calculating the loss gradient involves considerably less math and complexity - which I don't fully understand - and seems to accomplish the same thing:

$$Loss(w_0,w_1,w_2,...,b_0,b_1,b_2,...)$$

$$\nabla Loss(...)=\begin{bmatrix}{\frac{\partial Loss(...)}{\partial w_0}},{\frac{\partial Loss(...)}{\partial w_1}},...\end{bmatrix}$$

To figure out the gradient of this loss function, you can just tweak each parameter in the loss function and find out how much the loss changes. Dividing by how much you change it, you can get an approximation of the loss gradient at the current network state. Here is a rough outline of what I mean in code:

function LossGradient():
    current_loss = CalculateLoss()
    gradient_vector = []
    i = 0
    delta = 0.00001 # some really small value
    for weight in network:
        weight += delta # tweak weight by delta ( a tiny amount )
        new_loss = CalculateLoss() # calculate the loss after tweaking
        gradient_vector[i] = (new_loss - old_loss) / delta # store ∂Loss/∂W_i in gradient vector
        i += 1
    
    # do the same for biases

    return gradient_vector

My question is, what's the advantage of the backpropagation over this simple method that I used? I'm in the middle of struggling to understand the algorithm fully, and no resource seems to explain why and how exactly the algorithm is more efficient.

Any insight would be much appreciated.

Answer 1

If you know that $f(x) = wx + b$, then you also know that $\frac{\partial f}{\partial b} = 1$. The function LossGradient does not use that information, and would estimate it by evaluating the function $f$. Then it computes the loss. In general, it computes $f$ and $Loss$ once for every learnable parameter.

Backpropagation uses knowledge about the function being optimized to be more efficient. This means first evaluating $f$ once, then evaluating $Loss$ once, then updating all of your parameters. This efficiency may not matter to you when you have a small number of parameters, but it will make a huge difference if $f$ has hundreds of millions, or hundreds of billions of parameters.

It's not that an approach like yours is never useful (see derivative-free optimization), it's just that if you have extra information, using it is helpful.