As I know Gradient Descent has three variants which are:
1- Batch Gradient Descent: processes all the training examples for each iteration of gradient descent.
2- Stochastic Gradient Descent: processes 1 training example per iteration. Hence, the parameters are being updated even after one iteration in which only a single example has been processed.
3- Mini Batch gradient descent: which works faster than both batch gradient descent and stochastic gradient descent. Here b examples where b
But in some cases they use Stochastic Gradient Descent and they define a batch size for training, I am confused about that?
Also, what about Adam, AdaDelta, AdaGrad, are they all mini-batch gradient descent? or not?