2
$\begingroup$

I am currently studying the backpropagation process and gradient decent algorithm form the book Neural Networks and Deep Learning written by Michael Nielsen and 3Blue1Brown channel in YouTube.

My question is about calculating the gradient in gradient decent algorithm(the whole dataset as input).

I have drawn a picture that shows my understanding about how the algorithm works: enter image description here

For example we have 1 million hand written digit images and through the first iteration we feed the network with this 1 million images. Then the gradient is calculated for each images, summed together and averaged before updating the weights.

If my understanding is not wrong, this is the same thing that I saw in 3Blue1Brown channel. In this process the average of gradient is calculated with respect to the cost of each image and not the average cost of the whole data set(1 million) in one iteration, so the formula for calculating the cost has no effect here, rather its derivative is used for calculating the gradients for each image and we do not take the average of costs here.

First I want to know if this is a correct picture about how one iteration of calculating gradient decent works, second, why don't we take the derivatives of average costs with respect to the weights and biases and then take the average sum of all gradients?

And the last question, how the number of iterations and epochs can be assigned here? can we say number of epochs is always equal to the number of iteration since the whole data is used for each iteration?

Improve this question
$\endgroup$
2
  • $\begingroup$ $C_i$ stands for? $\endgroup$
    – hanugm
    Commented Nov 6, 2019 at 5:30
  • $\begingroup$ @hanugm The cost(error) of one image as input in one iteration $\endgroup$
    – Morti
    Commented Nov 6, 2019 at 19:57

1 Answer 1

Reset to default
0
$\begingroup$

Starting from the last part, as the entire dataset is used, number of epochs(run over entire dataset) equals number of iterations. Instead, one can do the calculation in "mini batches" (of 32, for example), then the run over each 32 samples is called an iteration.

As for the rest of the question, you can chose a batch that is equal to the entire dataset - this is called "batch gradient descent"; or update after every single sample (a batch size of 1) which is "stochastic gradient descent". Any other choice is called "mini-batch gradient descent.

Deep Learning course on Coursera offers a relatively better explanation of these matters compared to Nielsen's book or 3B1B videos. You can watch the videos for free. In particular here is the video on Gradient Descent.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the explaining of last part, but my question about the way I am picturing the gradient decent calculation has not answered yet. I want to know whether the gradient is calculated for each image and then taking the average sum or not. Also the part for average cost is another question. $\endgroup$
    – Morti
    Commented Nov 6, 2019 at 20:02
  • $\begingroup$ In addition to above, why do we update the batch size=1(entire dataset) in stochastic gradient descent? by "update" you mean shuffling only? because if we enter a whole new dataset, as far as I know that means online stochastic gradient decent. $\endgroup$
    – Morti
    Commented Nov 6, 2019 at 20:08
  • $\begingroup$ Summation and derivation can change places without changing the result for this calculation (it is a finite sum), so you can take the cost function or its derivative interchangeably. And batch size=1 does not mean the whole dataset, it is the exact opposite: each sample is taken as an individual batch, after with the update is made - which means changing the coefficients of the cost function (in another word, changing the cost function). $\endgroup$
    – serali
    Commented Nov 7, 2019 at 7:44
  • $\begingroup$ In think what you are trying to explain is the process of stochastic gradient decent and not gradient decent, my question above is about gradient decent(batch gradient decent) that the whole dataset feeds to the network. About the firts part in your answer, this thread says the result of taking average cost in the gradient decent is not equal to the average of gradient decent with each cost:datascience.stackexchange.com/questions/56405/… $\endgroup$
    – Morti
    Commented Nov 7, 2019 at 18:52
  • $\begingroup$ Looking from the comments/answers to that question, original poster may have tampered with the question after the replies. That question refers to SGD, in which update to the total cost function is made after the gradients of cost for each sample is calculated. What I was trying to say above is also stated in the replies given there, which is gradient of a sum is sum of gradients. But one can't carry this any further as suggested there, and sum over the variables of the function on which the gradient is applied. del(C(w1))+del(C(w2)) = del(C(w1)+C(w2)) but not del(C(w_1+w_2)) for nonlinear C. $\endgroup$
    – serali
    Commented Nov 8, 2019 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.