Backpropagation and Stochastic Gradient Descent(SGD)

Question

Is Backpropagation a learning method or an optimisation method?

How are Backpropagation and Stochastic Gradient Descent(SGD) related to each other?

Answer 1

Stochastic Gradient Descent (SGD) is an optimization method. As the name suggests, it depends on the gradient of the optimization objective.

Let's say you want to train a neural network. Usually, the loss function $L$ is defined as a mean or a sum over some "error" $l_i$ for each individual data point like this

$$ L(\theta) = \frac{1}{N} \sum_{i=0}^N l_i(\theta)$$

where $N$ is the number of data points and $\theta$ the model parameters.

For SGD you would randomly sample $i$ at each time step $t$ and do

$$ \theta_{t+1} = \theta_{t} - \alpha \nabla_{\theta}l_i(\theta_t)$$

with some learning rate $\alpha$ and the gradient with respect to the model parameters $\nabla_{\theta}$.

Backpropagation is now used to compute the gradient $\nabla_{\theta}l_i(\theta_t)$. As $l_i$ depends on a neural network with parameters $\theta$, this is not necessarily straight-forward, but it can be done quite efficiently using the chain rule in a smart way. This involves recursively computing the gradient of parameters in some layer using the gradients from higher layers, i.e. the gradients are computed starting at the network output and moving backwards. Hence the name backpropagation.

The wikipedia article on backpropagation goes through the math in a detailed manner.

Answer 2

I don't know that Backpropagation is a learning method or optimisation method but I know that Stochastic Gradient Descent(SGD) is not an optimisation method but the method of how to take data from dataset like Batch Gradient Descent(BGD) and Mini-Batch Gradient Descent(MBGD) to do Gradient Descent(GD) with the optimizers such as normal Gradient Descent(GD), Adam, RMSprop, Adadelta, Adagrad, etc.

• Backpropagation is the method which calculates a gradient using the average loss(difference) calculated by a loss function, working from output layer to input layer. Then, the gradient calculated by the Backpropagation is used to update model's parameters by Gradient Descent(GD) using the optimizer such as normal Gradient Descent(GD), Adam, RMSprop, Adadelta, etc to minimize the mean(average) of the sum of the losses(differences) between a model's predictions and true values(train data).

• SGD can do gradient descent with every single sample of a whole dataset one sample by one sample, taking the same number of steps as the samples of a whole dataset in one epoch. For example, a whole dataset has 100 samples(1x100), then gradient descent happens 100 times in one epoch which means model's parameters are updated 100 times in one epoch.