0
$\begingroup$

I am trying to reproduce results of other people's research, but we cannot afford to do it with the same batch size as theirs, due to limited computing resources.

The method they use is a simple sequence classification using BERT. They do it with batch size 48, learning rate 4e-5, optimization Adam, and epoch 1.

Their result is f2 score of 0.65 and we are nowhere near there despite doing everything the same except the batch size (we do it with batch size 16).

What are some ways to compensate for small batch size? I would greatly appreciate any suggestions.

$\endgroup$

1 Answer 1

0
$\begingroup$

Try looking into gradient accumulation, it is a method that instead of applying the gradient update after each batch adds up the gradients for multiple batches and applies them. This allows you to get the same gradient updates you would get with larger batches without needing the larger batches, which is often useful for situations where the model would need large batch sizes but you don't have enough computational resources to support those batch sizes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.