With a non-differentiable operation, such as a minimization, how does Tensorflow compute the gradients? Some kind of soft-minimum approximation? If so, can I retrieve the analytical computation for a specific gradient?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
A minimum operation is differentiable, or at least you can easily express the partial derivatives w.r.t. its inputs:
$f = min(x_1, x_2, x_3 ... x_n)$
$ \frac{\partial f}{\partial x_i} = \begin{cases} 1,& \text{if } argmin_i(x_i) = i\\ 0, & \text{otherwise} \end{cases}$
This does not hold strictly when multiple values share the same minimum value. However, that is not a problem in practice for gradient-based optimisers in TensorFlow, which can simply set all tied indices to have partial derivative of 1 (or a fraction $\frac{1}{n_{min}}$), with little impact to the eventual result, because ties for values will happen rarely. Ties may happen frequently enough in a ReLU-based network that the TensorFlow developers have considered a best response for them - I don't know specifically what TensorFlow does for that situation.
answered Aug 1, 2017 at 14:32
-
1$\begingroup$ Is there any reference to this in the Tensorflow documentation? $\endgroup$– learnerCommented Jan 14, 2020 at 5:03
-
$\begingroup$ Answer is contradictory. Says ~ "min is differentiable ... expect when it isn't ...". In a strict mathematical sense min(x,y,z,...) is not differentiable over it's domain. $\endgroup$– spinkusCommented Jan 14, 2020 at 5:41