Timeline for Why mini batch size is better than one single "batch" with all training data?
Current License: CC BY-SA 3.0
4 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 28, 2023 at 14:40 | comment | added | Antonios Sarikas | With respect to memory, for a GPU that can handle up to a mini-batch size of $B$, updates based on $b=1$ and $b=B$ will take the same amount of time, right (we take advantage of the parallelized computations)? If we exceed this threshold, e.g. $b=2B$, then the time for the update it doubles up. I wanted to add this comment, because it is generally stated that SGD is faster since it performs the gradient for only 1 example. Of course, this would be the case if we could not calculate the gradients on parallel, i.e. we would need one forward-backward pass for each sample in the batch. | |
| Dec 27, 2017 at 1:56 | history | edited | Peter | CC BY-SA 3.0 |
edited body
|
| Nov 6, 2017 at 8:27 | review | First posts | |||
| Nov 6, 2017 at 14:22 | |||||
| Nov 6, 2017 at 8:24 | history | answered | Peter | CC BY-SA 3.0 |