All Questions
Tagged with gradient neural-network
8 questions
0
votes
0
answers
14
views
How to derive the formula 13 in the Xavier Initialization paper
How to derive the formula 13 in the Xavier Initialization paper Understanding the difficulty of training deep feedforward neural networks from the formula 6?
- 791
0
votes
0
answers
149
views
calculating derivative of bias in backpropagation
Looking at the algorithm in wikipedia, we can implement backpropagation by calculating:
$$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$
(where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
- 101
1
vote
0
answers
55
views
How to interpret integrated gradients in an NLP toxic text classification use-case?
I am trying to understand how integrated gradients work in the NLP case.
Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
1
vote
0
answers
79
views
Which Neural Network or Gradient Boosting framework is the simplest for Custom Loss Functions?
I need to implement a custom loss function.
The function is relatively simple:
$$-\sum \limits_{i=1}^m [O_{1,i} \cdot y_i-1] \ \cdot \ \operatorname{ReLu}(O_{1,i} \cdot \hat{y_i} - 1)$$
With $O$ being ...
- 199
0
votes
1
answer
67
views
Vanishing gradient problem
In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers?
2
votes
1
answer
531
views
Gradient passthough in PyTorch
I need to quantize the inputs, but the method (bucketize) I need to do so is indifferentiable. I can of course detach the tensor, but then I lose the flow of gradients to earlier weights. I guess ...
- 203
4
votes
3
answers
12k
views
Forward pass vs backward pass vs backpropagation
As mentioned in the question, I have some issues understanding what are the differences between those terms.
From what I have understood:
Forward pass: compute the output of the network given the ...
2
votes
1
answer
905
views
What does it mean for a method to be invariant to diagonal rescaling of the gradients?
In the paper which describes Adam: a method for stochastic optimization, the author states:
The method is straightforward to implement, is computationally
efficient, has little memory requirements, ...
- 121