I am confused about the batch size of this model. I have used sgd i.e., Stochastic Gradient Descent as the optimizer (see the code). I am aware that in sgd, a single random instance from the training set is used to compute the gradient at each step. So, according to it, the batch_size should be equal to 1. Now, in the tf.keras.Sequential.fit() documentation it says:

If unspecified, batch_size will default to 32.

So, do I have to manually set the batch_size equal to 1? It is because the default value, 32 will make it a Mini-batch Gradient Descent.

    import tensorflow as tf
    from tensorflow import keras

    fashion_mnist = keras.datasets.fashion_mnist
    (X_train_full, y_train_full), (X_test, y_test) = fashion_mnist.load_data()

    X_valid, X_train = X_train_full[:5000]/255.0, X_train_full[5000:]/255.0
    y_valid, y_train = y_train_full[:5000], y_train_full[5000:]

    model = keras.models.Sequential()

    model.add(keras.layers.InputLayer(input_shape = [28, 28]))
    model.add(keras.layers.Dense(300, activation = "relu"))
    model.add(keras.layers.Dense(100, activation = "relu"))
    model.add(keras.layers.Dense(10, activation = "softmax"))

    model.compile(loss = "sparse_categorical_crossentropy", optimizer = "sgd", metrics = ["accuracy"])

    history = model.fit(X_train, y_train, epochs = 30, validation_data = (X_valid, y_valid))

Batch size specify the number of observations used to adjust the parameters for each iteration. If it is 1, the result from this observation will be used. If it is more than 1, average performance will be used.

Ideally you should consider batch size as a hyperparameter. Which means that you should determine the optimal batch size for your problem. You may use a simple for loop or grid/random search with other hyperparameters.

  • $\begingroup$ Actually, I was concerned that if I set my batch_size more than 1, will I still be able to call it a Stochastic Gradient Descent? According to my research, the answer should be no. So, I was wondering what is the point of using Stochastic Gradient Descent as the optimizer if I am not setting batch_size equal to 1. $\endgroup$ – Nowroz Islam Jul 18 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.