0
$\begingroup$

These days, training data aren't put in gradient descent all at once. Rather, they are put in batch after batch. Gradient descent is run once for each batch of training data. When all batches are traversed, 1 epoch is done. However, gradient descent hasn't minimized the loss function yet, the loss function still being on a slope. The loss function now and weights serve as initial values for the next epoch. Weights are initialized only at the beginning of the 1st batch of the 1st epoch.

During each epoch, the validation set is used to calculate the error (evaluated using trained weights) on the validation set. If the error is dropping, finish this epoch and go to the next epoch. Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.

If the above is all right, seems that the hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?

$\endgroup$
1
$\begingroup$

Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.

Typically, validation is done only at the end of the epoch. So no batch should remain unused. For example, this is the behavior of Keras https://keras.io/models/model/#fit

validation_freq: Only relevant if validation data is provided. Integer or list/tuple/set. If an integer, specifies how many training epochs to run before a new validation run is performed, e.g. validation_freq=2 runs validation every 2 epochs. If a list, tuple, or set, specifies the epochs on which to run validation, e.g. validation_freq=[1, 2, 10] runs validation at the end of the 1st, 2nd, and 10th epochs.

hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?

Yes, such hyper parameters are set when model is defined. These are not modified during training.

However : Systems like Google AutoML are trying to change this. These systems try to learn model hyper parameters during training.

| improve this answer | |
$\endgroup$
  • $\begingroup$ thx so much for correcting. but if validation is only done after a full epoch is finished, what's the point of sending training data batch by batch? when 1 batch goes thru gradient descent once, weights are updated once. if there's 10 batches in total, then gradient descent is gone thru 10 times in 1 epoch. if instead i put all the 10 batches together in, gradient descent is gone thru only once in this epoch. the result weight updates for these 2 scenarios r the same $\endgroup$ – feynman Mar 14 '19 at 8:05
  • 1
    $\begingroup$ 1. It is not possible to train 10 batches on GPUs . Typically GPU will have 4 - 16 GB memory. So, it does not have enough memory to load all data + model parameters $\endgroup$ – Shamit Verma Mar 14 '19 at 8:37
  • $\begingroup$ 2. The high througput may also translate to faster convergence depending on the variance in the dataset and the learning rate used. $\endgroup$ – Shamit Verma Mar 14 '19 at 8:37
  • 1
    $\begingroup$ 3. For smaller data sets, model can be trained without batching. $\endgroup$ – Shamit Verma Mar 14 '19 at 8:38
  • 1
    $\begingroup$ datascience.stackexchange.com/questions/16807/… $\endgroup$ – Shamit Verma Mar 14 '19 at 8:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.