Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Why do we think that stochastic gradient descent is going to find a minimum at all? I mean on each iteration SGD moves in the direction that reduces only current batch's error (SGD doesn't care about the rest of the samples). But why should this lead us to a local minimum of our cost function?
And why do we hope that this brand new minimum is going to be deeper than the initial one? Is it more likely and why? What is the reasoning?
For mini-batch gradient descent, the cost function may not decrease on every iteration. There is going to be some noise and smaller the batch size, noisier the process. SGD has batch size 1, so it is the extreme case. But still, an overall downward trend is to be expected. Compared to using entire dataset, SGD and mini-batch gradient descent is not going to converge to the minimum, but oscillate around the minimum. Check this video for a discussion of this process.
For mini-batches, we expect each batch to be a representative of the dataset; but batch size = 1 is too extreme. As the dataset should have some patterns to predict every sample would have some information within its data but as discussed above, we will have too much noise for a batch size of 1. In order to prevent this noise changing the function (the weights) too much for the next sample, the learning rate is set to a small value. In this way, even though we take a step in the wrong direction, it won't be a big step and the weights will be more or less the same as before for the next few samples. They will be different, but the difference will be small. It would be as if we were using a slightly larger batch size, plus some perturbation for the small differences in weights. This does not mean that smaller learning rate is always preferable, check this question to see how it should scale with batch size.
