2
$\begingroup$

From my understanding, a stochastic process whose value at a particular instant is dependent on the previous values taken and every time the process is run, the path chosen can be different. We can know only the limits and confines the process stays after certain intial seeding value.

The weight values for a neural networks are updated via the stochastic gradient descent method. What is the stochastic part about it? After some initialization of the variable, error function accumulated will be same if the same set of input data is provided after initialization in each test. Why is the relevance of the term stochastic? Where is the scope for the randomness and what is the source of this randomness?

|improve this question
$\endgroup$
4
$\begingroup$

You sample a random mini-batch as opposed to the full dataset. This means that you get a stochastic approximation of the true gradient of your loss function with regards to your dataset and weights.

|improve this answer
$\endgroup$
  • 1
    $\begingroup$ It is not just the sampling that is relevant to OP's question - it is the weight updates that occur after each mini-batch. If the gradients were accumulated in each mini-batch and then applied at the end of an epoch, then the process would be a batch one evaluated in parts. However, in mini-batch approach, once you have made a weight update on your randomly sampled approximation then you will take a different path each time (assuming different seed/init of RNG) $\endgroup$ – Neil Slater Sep 28 '17 at 11:15
  • $\begingroup$ So it's not stochastic gradient descent if input samples are not chosen randomly but rather the same fashion for every test run? And does the random initialization of weights count into the stochastic part? $\endgroup$ – Aditya Kumar Gupta Sep 28 '17 at 11:19
  • 1
    $\begingroup$ @AdityaKumarGupta: I expect that's just "Gradient Descent" if you work through examples methodically in order each epoch, calculating gradient and updating weights on each one. However, it would not behave the same as the batch method because you make a weight update on each example. Also, it is likely to suffer from bias compared to either batch GD or stochastic GD, so you won't see it used much $\endgroup$ – Neil Slater Sep 28 '17 at 11:24
  • 1
    $\begingroup$ @AdityaKumarGupta, not really. There are other ways to initialise weights, which are non-random. Jan van der Vegt has provided a very good answer. The "stochastic" part has to do with the fact that the minima where you converge is determined at random. $\endgroup$ – Digio Sep 28 '17 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.