Memory is not really the reason for doing this, because you could just accumulate your gradients as you iterate through the dataset, and apply them at the end, but still in SGD you apply them at every step.

Reasons that SGD is used so widely are:

1. Efficiency. Typically, especially early in training, the parameter-gradients for different subsets of the data will tend to point in the same direction. So gradients evaluated on 1/100th of the data will point roughly in the same general direction as on the full dataset, but only require 1/100 the computation. Since convergence on a highly-nonlinear deep network typically requires thousands or millions of iterations no matter how good your gradients are, it makes sense to do many updates based on cheap estimates of the gradient rather than few updates based on good ones.

2. Optimization: Noisy updates may allow you to bounce out of bad local optima (though I don't have a source that shows that this matters in practice).

3. Generalization. It seems (see Zhang et al: Theory of Deep Learning III: Generalization Properties of SGD) that SGD actually helps generalization by finding "flat" minima of the training set, which are more likely to also be minima of the test set. Intuitively, we can think of SGD as a sort of bootstrapping - by computing our parameters based on many subsets of the data, we're less prone to overfitting to the training set.

Memory is not really the reason for doing this, because you could just accumulate your gradients as you iterate through the dataset, and apply them at the end, but still in SGD you apply them at every step.

Reasons that SGD is used so widely are:

1. Efficiency. Typically, especially early in training, the parameter-gradients for different subsets of the data will tend to point in the same direction. So gradients evaluated on 1/100th of the data will point roughly in the same general direction as on the full dataset, but only require 1/100 the computation. Since convergence on a highly-nonlinear deep network typically requires thousands or millions of iterations no matter how good your gradients are, it makes sense to do many updates based on cheap estimates of the gradient rather than few updates based on good ones.

2. Optimization: Noisy updates may allow you to bounce out of bad local optima (though I don't have a source that shows that this matters in practice).

3. Generalization. It seems (see Zhang et al: Theory of Deep Learning III: Generalization Properties of SGD) that SGD actually helps generalization by finding "flat" minima on the training set, which are more likely to also be minima on the test set. Intuitively, we can think of SGD as a sort of Bagging - by computing our parameters based on many minibatches of the data, we reenforce rules that generalize across minibatches, and cancel rules that don't, thereby making us less prone to overfitting to the training set.