All Questions
7 questions
2
votes
0
answers
103
views
Derive backpropagation for PreLU
I want to derive the back propagation functions for the Parametric Relu activation function which is defined as follows:
$$
h_a(x) = \text{max}(ax, x)
$$
I want to derive $ \frac{\partial L}{\partial ...
- 21
1
vote
1
answer
1k
views
Problem with convergence of ReLu in MLP
I created neural network from scratch in python using only numpy and I'm playing with different activation functions. What I observed is quite weird and I would love to understand why this happens.
...
3
votes
1
answer
877
views
Why the sigmoid activation function results in sub-optimal gradient descent?
I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
- 33
1
vote
1
answer
1k
views
How does Pytorch deal with non-differentiable activation functions during backprop?
I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation ...
- 123
1
vote
0
answers
29
views
Wich activation function for DQL
After many research, I still can't find a neat answer about this question:
When I found the loss of my state-action pair. I'm only backpropagating that loss true the network and setting all other ...
1
vote
0
answers
273
views
Generalized softmax derivative for implementation with any loss function
I am currently taking some deep learning and neural network (NN) courses, and in addition to performing the course work, am implementing my own "toolkit" of NN techniques to better my understanding of ...
- 11
10
votes
1
answer
6k
views
Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training?
ReLU is an activation function defined as $h = \max(0, a)$ where $a = Wx + b$.
Normally, we train neural networks with first-order methods such as SGD, Adam, RMSprop, Adadelta, or Adagrad. ...
- 2,216