Questions tagged [gradient]
The gradient tag has no usage guidance.
29
questions
2
votes
1answer
39 views
Differentiable approximation for counting negative values in array
I have an array of time of arrivals and I want to convert it to count data using pytorch in a differentiable way.
Example arrival times:
...
0
votes
0answers
15 views
0
votes
0answers
18 views
Why is the signed gradient of the image used for adversarial examples [duplicate]
In this paper, the gradient of the loss w.r.t. to the image is computed, but its sign is used. Why is using the sign-method better?
1
vote
0answers
12 views
Central finite distance gradient simplified [closed]
I'm asked to compute central finite difference scheme (f(i+1)-f(i-1)) on an image. My attempt is something like:
...
1
vote
0answers
41 views
Which Neural Network or Gradient Boosting framework is the simplest for Custom Loss Functions?
I need to implement a custom loss function.
The function is relatively simple:
$$-\sum \limits_{i=1}^m [O_{1,i} \cdot y_i-1] \ \cdot \ \operatorname{ReLu}(O_{1,i} \cdot \hat{y_i} - 1)$$
With $O$ being ...
2
votes
1answer
957 views
How batch normalization layer resolve the vanishing gradient problem?
According to this article:
https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484
The vanishing gradient problem occurs when using the sigmoid ...
0
votes
0answers
8 views
Where was gradient accumulation originally proposed?
Gradient accumulation is a common technique when training neural network with large batch size that cannot fit into GPU memory. Yet I couldn't find where gradient accumulation was originally proposed. ...
0
votes
0answers
6 views
Gradient calculation of the pre-trained model
I have pre-trained tensorflow model in a graph format. I have not used tf.gradient on the graph structure.
In such cases, is there any way to calculate the gradient of some tensors operation?
0
votes
0answers
15 views
Gradients vanishing despite using Kaiming initialization
I was implementing a conv block in pytorch with activation function(prelu). I used Kaiming initilization to initialize all my weights and set all the bias to zero. However as I tested these blocks (by ...
0
votes
0answers
26 views
Gradient calculation analysis
I'm using VGG16 pretrained architecture for classification and visualization of result using Guided Backpropagation technique.
I have used tensorflow code for calaulating the input gradientSharing ...
0
votes
1answer
30 views
Vanishing gradient problem
In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers?
1
vote
1answer
98 views
Why does my manual derivative of Layer Normalization imply no gradient flow?
I recently tried computing the derivative of the layer norm function (https://arxiv.org/abs/1607.06450), an essential component of transformers, but the result suggests that no gradient flows through ...
0
votes
0answers
17 views
How can my loss be stable while the gradient keeps growing?
I have been working on an Offline/Batch Reinforcement Learning problem where I am using a BCQ-DDQN model as a Q-table. The model input is a state of 8 dimensions, and the output is a vector of Q-...
3
votes
3answers
899 views
Why a sign of gradient (plus or minus) is not enough for finding a steepest ascend?
Consider a simple 1-D function $y = x^2$ to find a maximum with the gradient ascent method.
If we start in point 3 on x-axis:
$$ \frac{\partial f}{\partial x} \...
1
vote
1answer
48 views
Gradient passthough in PyTorch
I need to quantize the inputs, but the method (bucketize) I need to do so is indifferentiable. I can of course detach the tensor, but then I lose the flow of gradients to earlier weights. I guess ...
1
vote
1answer
302 views
vanishing gradient and gradient zero
There is a well known problem vanishing gradient in BackPropagation training of ...
1
vote
0answers
31 views
How to choose appropriate epsilon value while approximating gradients to check training?
While approximating gradients, using actual epsilon to shift the weights results in wildly big gradient approximations, as the "width" of the used approximation triangle is ...
0
votes
1answer
81 views
implementing forward and backward of a Linear model
I'm implementing the code of this abstraction.
The forward is easy and looks like that:
I don't understand the backward path and how it fit's the abstraction in the first image:
Why is db defined as ...
0
votes
2answers
47 views
Intuitive explanation for representing gradient in higher dimensions
I do not understand how complex networks with many parameters/dimensions can be represented in a 3D space, and form a standard cost surface just like a simple network with, say, 2 parameters.
For ...
1
vote
1answer
84 views
Can mini-batch gradient descent outperform batch gradient descent? [duplicate]
As I was reading and going through the second course of Andrew Ng's deep learning course, I came across a sentence that said,
With a well-turned mini-batch size, usually it outperforms either
...
2
votes
1answer
825 views
Tensorflow.Keras: How to get gradient for an output class w.r.t a given input?
I have implemented and trained a sequential model using tf.keras. Say I am given an input array of size 8X8 and an output [0,1,0,...(rest all 0)].
How to calculate the gradient of the input w.r.t to ...
1
vote
1answer
69 views
CNN gradients with different magnitude
I have a CNN architecture with two cross entropy losses $\mathcal{L}_1$ and $\mathcal{L}_2$ summed in the total loss $\mathcal{L} = \mathcal{L}_1 + \mathcal{L}_2$. The task I want to solve is ...
0
votes
1answer
39 views
when x is a vector, derivative of vector diag(f'(x)) is formal notation?
https://web.stanford.edu/class/cs224n/readings/gradient-notes.pdf
(4)
this note says this
$$
\frac{\partial \textbf{z}}{\partial \textbf{x}} = \text{diag}(f'(\textbf{x}))
$$
I know this means make ...
0
votes
1answer
981 views
Gradient of a function in Python
I've defined a function in this way:
def qfun(par):
return(par[0]+atan(par[3])*par[1]+atan(par[4])*par[2])
How can I obtain the gradient of this function ...
0
votes
1answer
38 views
How can we get gradient with some other loss function apart from MSE?
In most of the gradient search examples, the update to weights are done by subtracting the derivative of MSE.
Can we have an example, where we did not use ...
1
vote
0answers
23 views
Matlab Optimization. Meaning of warning: "The slope should be 2. It appears to be 1."
I'm using the manopt package to solve some optimization problems in matlab. The problem is of the form.
problem.cost = @(x) f(x)
problem.egrad = @(x) g(x)
After the problem definition, I check ...
3
votes
1answer
59 views
Gradient Checking: MeanSquareError. Why huge epsilon improves discrepancy?
I am using custom C++ code, and coded a simple "Mean Squared Error" layer.
Temporarily using it for the 'classification task', not a simple regression. ...maybe this causes the issues?
I don't have ...
2
votes
1answer
96 views
Vanishing Gradient vs Exploding Gradient as Activation function?
ReLU is used as an activation function that serves two purposes:
Breaking linearity in DNN.
Helping in handling Vanishing Gradient problem.
For Exploding Gradient problem, we use Gradient Clipping ...
2
votes
1answer
627 views
What does it mean for a method to be invariant to diagonal rescaling of the gradients?
In the paper which describes Adam: a method for stochastic optimization, the author states:
The method is straightforward to implement, is computationally
efficient, has little memory requirements, ...
