Memory is *not* really the reason for doing this, because you *could* just accumulate your gradients as you iterate through the dataset, and apply them at the end, but still in SGD you apply them at every step.

Reasons that SGD is used so widely are:

1) Efficiency.  Typically, especially early in training, the parameter-gradients for different subsets of the data will tend to point in the same direction. So gradients evaluated on 1/100th of the data will point roughly in the same general direction as on the full dataset, but only require 1/100 the computation.  Since convergence on a highly-nonlinear deep network typically requires thousands or millions of iterations no matter how good your gradients are, it makes sense to do many updates based on cheap estimates of the gradient rather than few updates based on good ones.

2) Optimization: Noisy updates may allow you to bounce out of bad local optima (though I don't have a source that shows that this matters in practice).

3) Generalization.  It seems (see [Zhang et al: Theory of Deep Learning III: Generalization Properties of SGD](https://cbmm.mit.edu/sites/default/files/publications/CBMM-Memo-067.pdf)) that SGD actually helps generalization by finding "flat" minima on the training set, which are more likely to also be minima on the test set.  Intuitively, we can think of SGD as a sort of [Bagging](https://en.wikipedia.org/wiki/Bootstrap_aggregating) - by computing our parameters based on many minibatches of the data, we reenforce rules that generalize across minibatches, and cancel rules that don't, thereby making us less prone to overfitting to the training set.