1
$\begingroup$

I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation functions during backprop. However I was not able to come up with a full picture as to what exactly happens.

Most answers deal with ReLU $\max(0,1)$ and claims that the derivative at $0$ is either taken to be $0$ or $1$ by convention (not sure which one).

But there are many other activation functions with multiple points of non-differentiability.

enter image description here

2 points

enter image description here

4 points

How does Pytorch systematically deal with all these points during backprop? Does anyone have an authoritative answer?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

The function value is never exactly equal to those exact point because of numerical precision error.And again those functions in torch calculate left or right derivative which is defined in every case.So non-differentiability doesn't pose a problem here.

Improve this answer
$\endgroup$
1
  • $\begingroup$ That only covers the almost everywhere differentiable part of the question. Contrary to the OP it does make it much better, but they were not as creative on how bad you could make your Lambda layer. A more complete answer would say what sort of error or silent failure you get when you try to do such an absurd thing. $\endgroup$
    – AHusain
    Commented Feb 3 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.