Gradient clipping takes two main forms in Keras: gradient norm scaling (
clipnorm) and gradient value clipping (
1. Gradient Norm Scaling
Gradient norm scaling involves changing the derivatives of the loss function to have a given vector norm when the L2 vector norm (sum of the squared values) of the gradient vector exceeds a threshold value.
For example, we could specify a norm of 1.0, meaning that if the vector norm for a gradient exceeds 1.0, then the values in the vector will be rescaled so that the norm of the vector equals 1.0.
2. Gradient Value Clipping
Gradient value clipping involves clipping the derivatives of the loss function to have a given value if a gradient value is less than a negative threshold or more than the positive threshold. For example, we could specify a norm of 0.5, meaning that if a gradient value was less than -0.5, it is set to -0.5 and if it is more than 0.5, then it will be set to 0.5.
3. Keras Syntax
The gradient clipping syntax for Adaptive Moment Estimation (Adam) is very simple and follows the same syntax as for Stochastic Gradient Descent (SGD) shown above:
opt_adam = optimizers.adam(clipnorm=1.)
opt_adam = optimizers.adam(clipvalue=0.5)