I'm training a network using both learning rate warmup and an adaptive gradient clipping method outlined here. Is there a general consensus or anything in the literature relating to whether gradients should be clipped during the warmup phase or only after the warmup phase has completed.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.