Skip to main content

All Questions

Filter by
Sorted by
Tagged with
1 vote
1 answer
1k views

How does Pytorch deal with non-differentiable activation functions during backprop?

I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation ...
Norman's user avatar
  • 123
10 votes
1 answer
6k views

Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training?

ReLU is an activation function defined as $h = \max(0, a)$ where $a = Wx + b$. Normally, we train neural networks with first-order methods such as SGD, Adam, RMSprop, Adadelta, or Adagrad. ...
Rizky Luthfianto's user avatar