# How does Pytorch deal with non-differentiable activation functions during backprop?

I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation functions during backprop. However I was not able to come up with a full picture as to what exactly happens.

Most answers deal with ReLU $$\max(0,1)$$ and claims that the derivative at $$0$$ is either taken to be $$0$$ or $$1$$ by convention (not sure which one).

But there are many other activation functions with multiple points of non-differentiability.

2 points

4 points

How does Pytorch systematically deal with all these points during backprop? Does anyone have an authoritative answer?