All Questions
2 questions
10
votes
1
answer
6k
views
Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training?
ReLU is an activation function defined as $h = \max(0, a)$ where $a = Wx + b$.
Normally, we train neural networks with first-order methods such as SGD, Adam, RMSprop, Adadelta, or Adagrad. ...
- 107
1
vote
1
answer
1k
views
How does Pytorch deal with non-differentiable activation functions during backprop?
I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation ...
CommunityBot
- 1