5
$\begingroup$

When you .fit a Keras Sequential() model, you can specify a batch_size parameter. I have noteiced it is sometimes defined independently from the actual dataset size. Does it mean that mini-batches are sampled randomly, instead of scrolling down the dataset from top to bottom, slice by slice?

$\endgroup$

2 Answers 2

5
$\begingroup$

If you set shuffle=True as an argument of the model.fit method, Keras will shuffle the dataset before splitting it into batches (source), otherwise the dataset will be processed sequentially.

$\endgroup$
6
  • $\begingroup$ If you set a number of epochs and a batch size so that the number of observations are note enough... is the algorithm repeating from the start? $\endgroup$
    – Leevo
    Jun 18, 2019 at 7:04
  • $\begingroup$ what do you mean by "the number of observations are not enough"? if you have small batch size, you update the weights more frequently and vice versa. if you have more epochs, you go over the complete dataset more times. $\endgroup$
    – pcko1
    Jun 18, 2019 at 8:45
  • $\begingroup$ Let's say you have 1000 observations. If you set batch size = 100 and epochs = 20... does it start back from the top? $\endgroup$
    – Leevo
    Jun 18, 2019 at 8:47
  • 1
    $\begingroup$ it will restart 20 times from the top, because you have 20 epochs. And during each one of the 20 iterations, it will re-segment the data into batches of size 100. $\endgroup$
    – pcko1
    Jun 18, 2019 at 8:52
  • 1
    $\begingroup$ And when it restarts, does it make any reshuffle of the dataset, or always the same sequence of batches is fed into the Network? $\endgroup$
    – Leevo
    Jun 18, 2019 at 12:11
2
$\begingroup$

The relevant documentation doesn't mention random sampling per se.

NOTE: this all has nothing to do with the Sequential model type versus the Model type. OP was specifically talking about Sequential models.

You can specify the shuffle parameter to get random samples across the training dataset, but there is not a strict/parameterised sampling methodology. Using shuffle=True is however equiavalent to random selection without replacement (smaples can only be sampled once per epoch).

You can look through the source code to see how Keras builds up the train function, but it doesn't include any random sampling. This is taken care of deeper in the internals, via e.g. the fit_loop function, which simply shuffles the indices of the training samples:

if shuffle == 'batch':
    index_array = batch_shuffle(index_array, batch_size)
elif shuffle:
    np.random.shuffle(index_array)

You could pass class_weight argument to tell the Keras that some samples should be considered more important when computing the loss (although it doesn't affect the sampling method itself):

class_weight: Optional dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only). This can be useful to tell the model to "pay more attention" to samples from an under-represented class.

$\endgroup$
5
  • $\begingroup$ The class_weight affects the loss function, not the sampling probablity $\endgroup$
    – ignatius
    Jun 17, 2019 at 9:28
  • $\begingroup$ you mention "You can specify the shuffle parameter to get random samples across the training dataset." and then "see how Keras builds up the train function, but it doesn't include any random sampling", I think that's adversarial. $\endgroup$
    – pcko1
    Jun 17, 2019 at 9:32
  • 1
    $\begingroup$ I have updated to address comments - @pcko1 - I hope that is clearer now? $\endgroup$
    – n1k31t4
    Jun 17, 2019 at 9:51
  • $\begingroup$ Thanks for updating it but I think the message is still a bit confusing. You mention that random sampling is different than random shuffling, but in reality random.shuffle is equivalent to random.choice(replace=False), which samples numbers randomly. Long story short, np.random.shuffle will randomly shuffle your dataset. $\endgroup$
    – pcko1
    Jun 17, 2019 at 12:44
  • 1
    $\begingroup$ Good point - I just wanted to specoify that not smapling methods are really used, just shuffling the indices, but it is not parameterisable. I'll make another edit :) $\endgroup$
    – n1k31t4
    Jun 17, 2019 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.