0
$\begingroup$

I am trying to fine-tune a BERT model, but instead of doing it a fix number of training step, I want to use a stalling policy and allow it to run until the model stalls for N evaluations. However, I was previously using the transformers.get_linear_schedule_with_warmup, which requires an explicit number of training steps.

Is there any other learning rate scheduler that I should use for this task?

$\endgroup$

1 Answer 1

1
$\begingroup$

I don't know which framework to write out. But pytorch has torch.optim.lr_scheduler.ReduceLROnPlateau. This scheduler allows you to reduce the learning rate if the quality metric on the validation data has not changed for the better for more than n epochs (there are also other useful settings).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.