Timeline for Why mini batch size is better than one single "batch" with all training data?
Current License: CC BY-SA 3.0
16 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 10, 2022 at 23:25 | comment | added | partizanos | Why adding random noise to your data not have an equivalent effect ? (escaping saddle points) | |
| Dec 19, 2020 at 3:02 | comment | added | Green Falcon | @horaceT In your answer, you have referred to fundamental idea of stochastic gradient descent, but you have not talked about it. Do you have any idea for this | |
| Jun 16, 2020 at 11:08 | history | edited | CommunityBot |
Commonmark migration
|
|
| Apr 27, 2018 at 16:48 | history | edited | horaceT | CC BY-SA 3.0 |
added 209 characters in body
|
| Apr 27, 2018 at 5:21 | history | edited | horaceT | CC BY-SA 3.0 |
added 386 characters in body
|
| Apr 27, 2018 at 5:08 | history | edited | horaceT | CC BY-SA 3.0 |
added 764 characters in body
|
| Nov 6, 2017 at 8:29 | vote | accept | Hendrik | ||
| Oct 29, 2017 at 21:39 | comment | added | Wesley | @MartinThoma Given that there is one global minima for the dataset that we are given, the exact path to that global minima depends on different things for each GD method. For batch, the only stochastic aspect is the weights at initialization. The gradient path will be the same if you train the NN again with the same initial weights and dataset. For mini-batch and SGD, the path will have some stochastic aspects to it between each step from the stochastic sampling of data points for training at each step. This allows mini-batch and SGD to escape local optima if they are on the way. | |
| Feb 11, 2017 at 7:25 | comment | added | Martin Thoma | This paper is also on arXiv. Also, I don't see how this supports your claim. They never even mentioned mini-batch gradient descent. I do not understand that theorem (e.g. what is "g(X)"? Where did they introduce that notation? In statistics classes, g(X) = E(X)... but that doesn't make much sense here). What is $\phi(w, X)$? - The statement of this theorem seems to suggest that there are no bad local minima. But this would be true for SGD and batch gradient descent as well as mini-batch gradient descent, right? | |
| Feb 11, 2017 at 3:38 | history | edited | horaceT | CC BY-SA 3.0 |
edits
|
| Feb 11, 2017 at 3:17 | comment | added | horaceT | @MartinThoma See Theorem 6 in [2], a recent paper on JMLR. | |
| Feb 11, 2017 at 3:16 | history | edited | horaceT | CC BY-SA 3.0 |
added ref [2]
|
| Feb 9, 2017 at 21:00 | comment | added | Martin Thoma | Why should mini-batch gradient descent be more likely to avoid bad local minima than batch gradient descent? Do you have anything to back that claim up? | |
| Feb 7, 2017 at 20:57 | history | edited | horaceT | CC BY-SA 3.0 |
add more details
|
| Feb 7, 2017 at 20:44 | history | edited | horaceT | CC BY-SA 3.0 |
add ref
|
| Feb 7, 2017 at 20:29 | history | answered | horaceT | CC BY-SA 3.0 |