Is there a minimum batch size for training/re-fining a BERT model on custom data?

Could you name any cases where a mini batch size between 1-8 would make sense?

Would a batch size of 1 make sense at all?


Small mini-batch size leads to a big variance in the gradients. In theory, with a sufficiently small learning rate, you can learn anything even with very small batches.

In practice, Transformers are known to work best with very large batches. You can simulate large batches by accumulating gradients from the mini-batches and only do the update once in several steps.

Also, when finetuning BERT, you might also think of fine-tuning only the last layer (or several last layers), so you save some memory on the parameter gradients and can have bigger batches.

  • $\begingroup$ Thanks! Will tag as the correct answer! Could you please provide an example for ‘accumulating gradients from the mini-batches and update once in several steps’? $\endgroup$ – Predicted Life Nov 27 '20 at 10:32
  • $\begingroup$ In PyTorch, it is done easily by calling loss.backward() after each batch (the gradients get added to the existing ones) and calling optimizer.step() with optimizer.zero_grad() only in every k-th step. $\endgroup$ – Jindřich Nov 27 '20 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.