I am replicating, in Keras, the work of a paper where I know the values of epoch and batch_size. Since the dataset is quite large, I am using fit_generator. I would like to know what to set in steps_per_epoch given epoch value and batch_size. Is there a standard way?
$\begingroup$
$\endgroup$
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
7
As mentioned in Keras' webpage about fit_generator():
steps_per_epoch: Integer. Total number of steps (batches of samples) to yield from generator before declaring one epoch finished and starting the next epoch. It should typically be equal to ceil(num_samples / batch_size). Optional for Sequence: if unspecified, will use the len(generator) as a number of steps.
You can set it equal to num_samples // batch_size, which is a typical choice.
However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.
Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.
-
$\begingroup$ I saw the documentation before posting the question, I couldn't understand what it means by
num_samples. Does it mean total number of rows inx_trainfile? $\endgroup$ Commented Mar 16, 2019 at 14:29 -
$\begingroup$ The learning rate is decayed at each step, not at each epoch, according to github.com/keras-team/keras/blob/… $\endgroup$– BananachCommented Feb 17, 2020 at 18:03
-
$\begingroup$ @Bananach I believe we are saying the same thing, right? $\endgroup$– pcko1Commented Feb 17, 2020 at 21:54
-
$\begingroup$ I didn't think so. You are
saying steps_per_epochallows for the learning rate to be adjusted during a pass through the samples, bysetting steps_per_epoch < num_samples//batch_size. I am saying that that'll happen anyway even withsteps_per_epoch=num_samples//batch_size, and it'll happen exactlysteps_per_epochoften $\endgroup$– BananachCommented Feb 17, 2020 at 22:09 -
1$\begingroup$ The real question is : after steps_per_epoch batches have passed and the epoch is considered finished, will the next epoch start again from 1st batch of the dataset or will it continue from the next batch ? If it starts again at first batch, this is an issue as some batches in the dataset will never be used. $\endgroup$ Commented Apr 6, 2021 at 10:04
$\begingroup$
$\endgroup$
2
I think it would be nice to have the following relation hold
steps_per_epoch * batch_size = number_of_rows_in_train_data
This will result in usage of all the train data for one epoch.
Also, consider using fit() instead of fit_generator() if you need to have fast performance, but take into account that fit() might use more memory.
-
$\begingroup$ I think that relation does hold. At least, that's implied here: pyimagesearch.com/2018/12/24/… $\endgroup$ Commented Aug 20, 2019 at 15:52
-
$\begingroup$ @StatsSorceress: This is only true when the number_of_rows_in_train_data is divisible by batch_size. Otherwise, I assume, the remaining data is unused. $\endgroup$ Commented Feb 6, 2020 at 9:05
$\begingroup$
$\endgroup$
For example, if you have 100 training samples, then num_samples = 100, or the number of rows of x_train is 100.
You can specify your own batch size. In this case, say batch_size = 20. As a result, you can set your steps_per_epoch = 100/20 = 5 because in this way you can make use of the complete training data for each epoch.
If you also want to ask the scenario you want to set steps_per_epoch != num_samples/batch_size (for example, when num_samples cannot be fully divided by batch_size), please refer to this post: https://github.com/keras-team/keras/issues/10164
$\begingroup$
$\endgroup$
Let's clear it :
Assume you have a dataset with 8000 samples (rows of data) and you choose a batch_size = 32 and epochs = 25
This means that the dataset will be divided into (8000/32) = 250 batches, having 32 samples/rows in each batch. The model weights will be updated after each batch.
one epoch will train 250 batches or 250 updations to the model.
here steps_per_epoch = no.of batches
With 50 epochs, the model will pass through the whole dataset 50 times.
Ref - https://machinelearningmastery.com/difference-between-a-batch-and-an-epoch/
$\begingroup$
$\endgroup$
This is the formal definition : It should typically be equal to the number of unique samples of your dataset divided by the batch size.
This is how it works,
When you provide 's' steps per epoch , Each 's' step will have 'x' batches each consisting 'n' samples are sent to fit_generator,
So, if you specify 5 steps per epoch, each epoch computes 'x' batches each consisting of 'n' samples 5 times, then the next epoch is started!
