# What is the stochastic part in stochastic gradient descent?

From my understanding, a stochastic process whose value at a particular instant is dependent on the previous values taken and every time the process is run, the path chosen can be different. We can know only the limits and confines the process stays after certain intial seeding value.

The weight values for a neural networks are updated via the stochastic gradient descent method. What is the stochastic part about it? After some initialization of the variable, error function accumulated will be same if the same set of input data is provided after initialization in each test. Why is the relevance of the term stochastic? Where is the scope for the randomness and what is the source of this randomness?

You sample a random mini-batch as opposed to the full dataset. This means that you get a stochastic approximation of the true gradient of your loss function with regards to your dataset and weights.

• It is not just the sampling that is relevant to OP's question - it is the weight updates that occur after each mini-batch. If the gradients were accumulated in each mini-batch and then applied at the end of an epoch, then the process would be a batch one evaluated in parts. However, in mini-batch approach, once you have made a weight update on your randomly sampled approximation then you will take a different path each time (assuming different seed/init of RNG) Commented Sep 28, 2017 at 11:15
• So it's not stochastic gradient descent if input samples are not chosen randomly but rather the same fashion for every test run? And does the random initialization of weights count into the stochastic part? Commented Sep 28, 2017 at 11:19
• @AdityaKumarGupta: I expect that's just "Gradient Descent" if you work through examples methodically in order each epoch, calculating gradient and updating weights on each one. However, it would not behave the same as the batch method because you make a weight update on each example. Also, it is likely to suffer from bias compared to either batch GD or stochastic GD, so you won't see it used much Commented Sep 28, 2017 at 11:24
• @AdityaKumarGupta, not really. There are other ways to initialise weights, which are non-random. Jan van der Vegt has provided a very good answer. The "stochastic" part has to do with the fact that the minima where you converge is determined at random. Commented Sep 28, 2017 at 12:30

In the context of stochastic gradient descent (SGD) in neural networks, the term "stochastic" refers to the randomness introduced during the weight updates. Unlike traditional gradient descent, which computes the gradient using the entire dataset, SGD updates the weights based on a random subset of the training data, known as a mini-batch. This introduces stochasticity or randomness into the optimization process.

The randomness arises from the random selection of mini-batches from the training data. Each mini-batch represents a random sample of the dataset, and the gradients computed on these mini-batches are used to update the weights. By randomly sampling mini-batches, SGD introduces variations in the estimated gradients, which can help the optimization process escape local minima and explore different regions of the weight space.