# What should be the value of batch_size in fit() method when using sgd (Stochastic Gradient Descent) as the optimizer?

I am confused about the batch size of this model. I have used sgd i.e., Stochastic Gradient Descent as the optimizer (see the code). I am aware that in sgd, a single random instance from the training set is used to compute the gradient at each step. So, according to it, the batch_size should be equal to 1. Now, in the tf.keras.Sequential.fit() documentation it says:

If unspecified, batch_size will default to 32.

So, do I have to manually set the batch_size equal to 1? It is because the default value, 32 will make it a Mini-batch Gradient Descent.

    import tensorflow as tf
from tensorflow import keras

fashion_mnist = keras.datasets.fashion_mnist
(X_train_full, y_train_full), (X_test, y_test) = fashion_mnist.load_data()

X_valid, X_train = X_train_full[:5000]/255.0, X_train_full[5000:]/255.0
y_valid, y_train = y_train_full[:5000], y_train_full[5000:]

model = keras.models.Sequential()

model.compile(loss = "sparse_categorical_crossentropy", optimizer = "sgd", metrics = ["accuracy"])

history = model.fit(X_train, y_train, epochs = 30, validation_data = (X_valid, y_valid))


First, using the appropriate terminology you can say batch Stochastic Gradient Descent and batch Gradient descent are in the extreme ends, where Stochastic Gradient Descent is training with $$batch size = 1$$ and for batch gradient descent with $$batch size=n$$ where $$n$$ denotes the number of data points.
In the appropriate terminology, what we are often using (and similar to your example as well) is called mini-batch gradient descent. Note that the term mini here does not mean it is necessarily very small like 4, 32 or 64 but instead can be anything bigger than $$1$$ but smaller than $$n$$. In practice, people use the term mini-batch gradient descent and stochastic gradient descent interchangeably. This is because in practice they behave similarly.
• Actually, I was concerned that if I set my batch_size more than 1, will I still be able to call it a Stochastic Gradient Descent? According to my research, the answer should be no. So, I was wondering what is the point of using Stochastic Gradient Descent as the optimizer if I am not setting batch_size equal to 1. Jul 18 '19 at 17:10