1
$\begingroup$

I am confused about the batch size of this model. I have used sgd i.e., Stochastic Gradient Descent as the optimizer (see the code). I am aware that in sgd, a single random instance from the training set is used to compute the gradient at each step. So, according to it, the batch_size should be equal to 1. Now, in the tf.keras.Sequential.fit() documentation it says:

If unspecified, batch_size will default to 32.

So, do I have to manually set the batch_size equal to 1? It is because the default value, 32 will make it a Mini-batch Gradient Descent.

    import tensorflow as tf
    from tensorflow import keras

    fashion_mnist = keras.datasets.fashion_mnist
    (X_train_full, y_train_full), (X_test, y_test) = fashion_mnist.load_data()

    X_valid, X_train = X_train_full[:5000]/255.0, X_train_full[5000:]/255.0
    y_valid, y_train = y_train_full[:5000], y_train_full[5000:]

    model = keras.models.Sequential()

    model.add(keras.layers.InputLayer(input_shape = [28, 28]))
    model.add(keras.layers.Flatten())
    model.add(keras.layers.Dense(300, activation = "relu"))
    model.add(keras.layers.Dense(100, activation = "relu"))
    model.add(keras.layers.Dense(10, activation = "softmax"))

    model.compile(loss = "sparse_categorical_crossentropy", optimizer = "sgd", metrics = ["accuracy"])

    history = model.fit(X_train, y_train, epochs = 30, validation_data = (X_valid, y_valid))
$\endgroup$
0
$\begingroup$

First, using the appropriate terminology you can say batch Stochastic Gradient Descent and batch Gradient descent are in the extreme ends, where Stochastic Gradient Descent is training with $batch size = 1$ and for batch gradient descent with $batch size=n$ where $n$ denotes the number of data points.

In the appropriate terminology, what we are often using (and similar to your example as well) is called mini-batch gradient descent. Note that the term mini here does not mean it is necessarily very small like 4, 32 or 64 but instead can be anything bigger than $1$ but smaller than $n$. In practice, people use the term mini-batch gradient descent and stochastic gradient descent interchangeably. This is because in practice they behave similarly.

I personally do not think that such practice (using SGD and minibatch SGD interchangeably) is bad, because I don't think that it differs a lot such that it requires a specific new term.

$\endgroup$
0
$\begingroup$

Batch size specify the number of observations used to adjust the parameters for each iteration. If it is 1, the result from this observation will be used. If it is more than 1, average performance will be used.

Ideally you should consider batch size as a hyperparameter. Which means that you should determine the optimal batch size for your problem. You may use a simple for loop or grid/random search with other hyperparameters.

$\endgroup$
  • 1
    $\begingroup$ Actually, I was concerned that if I set my batch_size more than 1, will I still be able to call it a Stochastic Gradient Descent? According to my research, the answer should be no. So, I was wondering what is the point of using Stochastic Gradient Descent as the optimizer if I am not setting batch_size equal to 1. $\endgroup$ – Nowroz Islam Jul 18 '19 at 17:10
  • 1
    $\begingroup$ Its called Mini-batch stochastic gradient descent , very detailed explanation is provided in the below link. stats.stackexchange.com/questions/140811/… $\endgroup$ – Saandeep Sreerambatla Dec 20 '19 at 6:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.