5
$\begingroup$

With a non-differentiable operation, such as a minimization, how does Tensorflow compute the gradients? Some kind of soft-minimum approximation? If so, can I retrieve the analytical computation for a specific gradient?

$\endgroup$
2
$\begingroup$

A minimum operation is differentiable, or at least you can easily express the partial derivatives w.r.t. its inputs:

$f = min(x_1, x_2, x_3 ... x_n)$

$ \frac{\partial f}{\partial x_i} = \begin{cases} 1,& \text{if } argmin_i(x_i) = i\\ 0, & \text{otherwise} \end{cases}$

This does not hold strictly when multiple values share the same minimum value. However, that is not a problem in practice for gradient-based optimisers in TensorFlow, which can simply set all tied indices to have partial derivative of 1 (or a fraction $\frac{1}{n_{min}}$), with little impact to the eventual result, because ties for values will happen rarely. Ties may happen frequently enough in a ReLU-based network that the TensorFlow developers have considered a best response for them - I don't know specifically what TensorFlow does for that situation.

$\endgroup$
  • $\begingroup$ Is there any reference to this in the Tensorflow documentation? $\endgroup$ – learner Jan 14 at 5:03
  • $\begingroup$ Answer is contradictory. Says ~ "min is differentiable ... expect when it isn't ...". In a strict mathematical sense min(x,y,z,...) is not differentiable over it's domain. $\endgroup$ – spinkus Jan 14 at 5:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.