I am trying to understand the difference between validation steps and steps per epoch, Can anybody tell me the difference between these two terms. I also want to know about, how will it be helpful in training and what number should we set in it?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
Validation steps
While training, a machine learning model performs training steps on the training data. A step is a single backpropagation performed by the model. Similarly, the validation step is a single backpropagation performed on the validation data. The validation steps are generally performed after several training steps to track the validation accuracy. This way, we can track the model during its training whether it is overfitting or not.
Steps per epoch
Once we know what is a step, the next term is how many steps does a model will perform in a single epoch. We call it as one epoch when model has performed several steps and completed the whole training data. In the next epoch, the model will begin its steps again from the start of the training data. And while training it using the stochastic gradient descent process (it is generally used to train the deep learning models) we need to specify a batch size. For eg - if we specify the batch size as 16, the model will perform 16 forward propagations, then the loss will be calculated on all 16 predictions that the model has predicted and only one back propagation will be performed to correct the weights. And let's say there are 1600 training examples in the training set, so there will be 1600/16 = 100 batches. Also, there will be 100 backpropagations. 1 backpropagation = 1 step. Thus, there are 100 steps per epoch for this dataset.
Choosing numbers for both of them
Validation steps cannot be chosen. You should perform all the steps on the validation set and track It as your validation accuracy (or relevant metric). If you mean, you want to choose a number of steps after which validation steps should be performed, then that totally depends on your speed and convenience. This won't affect the performance of the model directly.
Steps per epoch will directly affect the performance of the model. And, it depends on the batch size. This blog will help you understand how batch sizes affect the training process. And (steps per epoch) = (total number of training instances) / (batch size).
-
$\begingroup$ I think there is some misconception in steps per epoch , because You said that if we set the batch size of 16 then 16 forward propagations performed . But I read that model takes the single pass to performed that iteration , and then loss calculated , i don't think that 16 forwards passes will be done! $\endgroup$– HamzaCommented Jun 27, 2021 at 9:59
-
$\begingroup$ What is difference between steps and iterations per epochs , if 1600/16 = 100 batches and 100 iterations are required to complete one epochs then how steps and iterations are different ? $\endgroup$– HamzaCommented Jun 27, 2021 at 10:14
-
$\begingroup$ @Hamza, We call it as batch inference. Model performs 16 parallel forward props using vectorization and not sequentially one by one. In batch inference a single batch is loaded on the device (CPU, GPU or TPU) and low level optimizations are used to perform super fast inference of all 16 inputs. On the other hand, normally a model can also perform 16 forward props sequentially, but it is not an efficient way to do so. The model actually performs 16 forward props in both the cases. Similarly the loss is calculated for the whole batch. And this loss is used to for a single backprop. $\endgroup$ Commented Jun 27, 2021 at 16:31
-
$\begingroup$ Steps and iterations can be used interchangeably in this context. That is, step and iteration mean the same thing. $\endgroup$ Commented Jun 27, 2021 at 16:33