I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation functions during backprop. However I was not able to come up with a full picture as to what exactly happens.
Most answers deal with ReLU $\max(0,1)$ and claims that the derivative at $0$ is either taken to be $0$ or $1$ by convention (not sure which one).
But there are many other activation functions with multiple points of non-differentiability.
How does Pytorch systematically deal with all these points during backprop? Does anyone have an authoritative answer?