Timeline for Calculating the average of gradient decent
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 8, 2019 at 20:30 | vote | accept | Morti | ||
| Nov 8, 2019 at 10:59 | comment | added | serali | Looking from the comments/answers to that question, original poster may have tampered with the question after the replies. That question refers to SGD, in which update to the total cost function is made after the gradients of cost for each sample is calculated. What I was trying to say above is also stated in the replies given there, which is gradient of a sum is sum of gradients. But one can't carry this any further as suggested there, and sum over the variables of the function on which the gradient is applied. del(C(w1))+del(C(w2)) = del(C(w1)+C(w2)) but not del(C(w_1+w_2)) for nonlinear C. | |
| Nov 7, 2019 at 18:52 | comment | added | Morti | In think what you are trying to explain is the process of stochastic gradient decent and not gradient decent, my question above is about gradient decent(batch gradient decent) that the whole dataset feeds to the network. About the firts part in your answer, this thread says the result of taking average cost in the gradient decent is not equal to the average of gradient decent with each cost:datascience.stackexchange.com/questions/56405/… | |
| Nov 7, 2019 at 7:44 | comment | added | serali | Summation and derivation can change places without changing the result for this calculation (it is a finite sum), so you can take the cost function or its derivative interchangeably. And batch size=1 does not mean the whole dataset, it is the exact opposite: each sample is taken as an individual batch, after with the update is made - which means changing the coefficients of the cost function (in another word, changing the cost function). | |
| Nov 6, 2019 at 20:08 | comment | added | Morti | In addition to above, why do we update the batch size=1(entire dataset) in stochastic gradient descent? by "update" you mean shuffling only? because if we enter a whole new dataset, as far as I know that means online stochastic gradient decent. | |
| Nov 6, 2019 at 20:02 | comment | added | Morti | Thanks for the explaining of last part, but my question about the way I am picturing the gradient decent calculation has not answered yet. I want to know whether the gradient is calculated for each image and then taking the average sum or not. Also the part for average cost is another question. | |
| Nov 6, 2019 at 8:09 | history | answered | serali | CC BY-SA 4.0 |