Back-propagation and stochastic gradient descent

Question

Is backpropagation a learning method or an optimisation method?

How are backpropagation and stochastic gradient descent related to each other?

Answer 1

Stochastic Gradient Descent (SGD) is an optimization method. As the name suggests, it depends on the gradient of the optimization objective.

Let's say you want to train a neural network. Usually, the loss function $L$ is defined as a mean or a sum over some "error" $l_i$ for each individual data point like this

$$ L(\theta) = \frac{1}{N} \sum_{i=0}^N l_i(\theta)$$

where $N$ is the number of data points and $\theta$ the model parameters.

For SGD you would randomly sample $i$ at each time step $t$ and do

$$ \theta_{t+1} = \theta_{t} - \alpha \nabla_{\theta}l_i(\theta_t)$$

with some learning rate $\alpha$ and the gradient with respect to the model parameters $\nabla_{\theta}$.

Backpropagation is now used to compute the gradient $\nabla_{\theta}l_i(\theta_t)$. As $l_i$ depends on a neural network with parameters $\theta$, this is not necessarily straight-forward, but it can be done quite efficiently using the chain rule in a smart way. This involves recursively computing the gradient of parameters in some layer using the gradients from higher layers, i.e. the gradients are computed starting at the network output and moving backwards. Hence the name backpropagation.

The wikipedia article on backpropagation goes through the math in a detailed manner.