I recently proved that
$\mathbb{E}_I[\nabla L_I(x,y,w)] = \nabla(L_I(x,y,w))$
Where $I$ represents a randomized batch of size $m$.
However, I cannot really understand why this is good news. I know that gradient is important for optimizing a model, especially with gradient descent. I also understand that in mini-batch algorithms we average over a whole batch instead of doing it for all the data, saving a lot of computations and reducing the variance.
It is usually good news when an approximating function you use to take an action to optimise something is unbiased compared to the thing being approximated. It means your actions in aggregate will head towards the same optimisation point.
If the gradient was biased in the long term, then (depending on nature of the bias) an optimisation routine will converge incorrectly by following them in gradient descent, and give an incorrectly optimised answer.
It is sometimes OK if a value is initially biased, but the bias reduces over time towards zero. Not that this is the case here, but you will see that in some machine learning algorithms. For instance, in a neural network using momentum, typically the initial momentum values are biased, but the bias decays exponentially (it is actually possible to make an unbiased momentum estimate, but most implementations of SGD with momentum or Nesterov momentum do not do that - Adam implementations typically do).
