0
$\begingroup$

These days, training data aren't put in gradient descent all at once. Rather, they are put in batch after batch. Gradient descent is run once for each batch of training data. When all batches are traversed, 1 epoch is done. However, gradient descent hasn't minimized the loss function yet, the loss function still being on a slope. The loss function now and weights serve as initial values for the next epoch. Weights are initialized only at the beginning of the 1st batch of the 1st epoch.

During each epoch, the validation set is used to calculate the error (evaluated using trained weights) on the validation set. If the error is dropping, finish this epoch and go to the next epoch. Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.

If the above is all right, seems that the hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.

Typically, validation is done only at the end of the epoch. So no batch should remain unused. For example, this is the behavior of Keras https://keras.io/models/model/#fit

validation_freq: Only relevant if validation data is provided. Integer or list/tuple/set. If an integer, specifies how many training epochs to run before a new validation run is performed, e.g. validation_freq=2 runs validation every 2 epochs. If a list, tuple, or set, specifies the epochs on which to run validation, e.g. validation_freq=[1, 2, 10] runs validation at the end of the 1st, 2nd, and 10th epochs.

hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?

Yes, such hyper parameters are set when model is defined. These are not modified during training.

However : Systems like Google AutoML are trying to change this. These systems try to learn model hyper parameters during training.

Improve this answer
$\endgroup$
7
  • $\begingroup$ thx so much for correcting. but if validation is only done after a full epoch is finished, what's the point of sending training data batch by batch? when 1 batch goes thru gradient descent once, weights are updated once. if there's 10 batches in total, then gradient descent is gone thru 10 times in 1 epoch. if instead i put all the 10 batches together in, gradient descent is gone thru only once in this epoch. the result weight updates for these 2 scenarios r the same $\endgroup$
    – feynman
    Commented Mar 14, 2019 at 8:05
  • 1
    $\begingroup$ 1. It is not possible to train 10 batches on GPUs . Typically GPU will have 4 - 16 GB memory. So, it does not have enough memory to load all data + model parameters $\endgroup$
    – Shamit Verma
    Commented Mar 14, 2019 at 8:37
  • $\begingroup$ 2. The high througput may also translate to faster convergence depending on the variance in the dataset and the learning rate used. $\endgroup$
    – Shamit Verma
    Commented Mar 14, 2019 at 8:37
  • 1
    $\begingroup$ 3. For smaller data sets, model can be trained without batching. $\endgroup$
    – Shamit Verma
    Commented Mar 14, 2019 at 8:38
  • 1
    $\begingroup$ datascience.stackexchange.com/questions/16807/… $\endgroup$
    – Shamit Verma
    Commented Mar 14, 2019 at 8:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.