2
$\begingroup$

I just want to clear one doubt - we use gradient descent to optimize the weights and biases of the neural network, and we use backpropagation for the step that requires calculating partial derivatives of the loss function, or am I misinterpreting something?

$\endgroup$
1
  • $\begingroup$ Yes, you got it right. To compute the gradients ($\partial loss/\partial w,\,\,\partial loss/\partial b$) we use backpropagation. And in order to compute the updates of weights ($w$) and biases ($b$) we can make use of this gradient (and its variants) $\rightarrow \delta w = \alpha \frac{\partial loss}{\partial w}$, $\,\,\delta b = \alpha \frac{\partial loss}{\partial b}$ $\endgroup$
    – Javier TG
    Oct 15 '20 at 16:16
1
$\begingroup$

Yes you are correct. Gradient descent (or various flavors of it) is the mechanism by which you find a local minima of your loss space using some learning rate. Backpropagation calculates the gradient of the error function with respect to the NN's weights and biases.

$\endgroup$
0
$\begingroup$

We have an ANN defined:

$$A = (N, I_n, P, g)$$

$$N: ℝ^{|X_i|} ℝ^{|I_n|} \rightarrow ℝ^{|O_n|}, ~~ N(X_i,I_n) = O^i_n$$

With epoch $E=E_X\cup E_Y$ of input records $E_X = \{X_0,...,X_n\}$ and desired outputs $E_Y = \{Y_0,...,Y_n\}$, a weight-bias vector $I_n$ trained $n$ times, a set of neurons, $P$, and an activation function $g$. The bias can either be different for each neuron, different for each layer or a single one for the network.

We have a neuron input for the $j$th neuron on layer $i$: $$a^i_j = \sum_l {w_j}^i_l ~ z^{i-1}_l$$

We have a neuron output for the $j$th neuron on layer $i$: $$z^i_j =\begin{cases}g(a^i_j) & i\in\mathbb{Z^+}\\ x^i_j & i=0 \\ \end{cases}$$ Where $g$ is the activation function

We have an initial weight-bias vector: $$I_0 = (w_0,...,w_n, b_0,...,b_n)$$

We have an initial set of outputs for record $X_0$: $$N(X_0, I_0) = O^0_0 = \{o_0,...,o_n\}$$

We have a desired set of outputs: $$Y_0 = \{y_1,...,y_n\}$$

We have a cost function:

$$C_i: ℝ^{I_n} \rightarrow ℝ, ~~ C_i(I_n) = \frac{1}{|N(X_i,I_n)|} \sum_{o,y~\in~N(X_i,I_n), Y_n}(o - y)^2$$

$$\nabla C_i(I_n) = \left(\frac{\partial C_i}{\partial w}, \frac{\partial C_i}{\partial b} \right)$$

Where $w = w_0,...,w_n$ and $l$ is the learning rate. Note that each weight and bias is involved in the cost function, because it computes over $o_i$ which are formed from the weights and biases in the network, therefore each one has a partial derivative with respect to the cost function that can be worked out through backpropagation, which is just using the chain rule to work out those partial derivatives in the gradient vector of the cost function $\nabla C_i$ at $I_n$. This is indeed done by realising the output is the function (given by some activation function) of the outputs of the previous layer multiplied by their weights plus a bias.

We have batch gradient descent (which gives a new weight-bias vector by computing an error vector as the sum of the mean square errors from all the test input records and subtracting the error vector from the current weight-bias vector):

$$I_{n+1} = I_n - l\sum_{i=0}^{|E_X|-1} \nabla C_i(I_n)$$

We have a stochastic gradient descent (which gives a new weight-bias vector by computing an error vector as the mean square error of a single test input records and subtracting the error vector from the current weight-bias vector):

$$I_{n+1} = I_n - l\nabla C_n(I_n)$$

As you keep performing the recurrence relation, you should get closer to a local/global minimum. You move in the direction of the gradient of steepest descent on the error surface at the current location $I_n$, and you are locked to the surface -- picture $z=f(x,y)$, you can only change $x$ and $y$ and each $(x,y)$ has a $z$ coordinate which places it on the surface.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.