Questions tagged [gradient]
The gradient tag has no usage guidance.
36 questions
0
votes
0
answers
10
views
How to make zero padding gradually smoother for continous data?
I have this continous data (amplitude):
I want to make gradually smoother for zero padding, actually I succeded using convolution. But it requires hard tuning and I'm not sure it's applied for all ...
0
votes
0
answers
330
views
NaN grad norm even with a stable loss and gradient
Currently I am working on a custom fine-tune of several code LLMs and while working on the DeepSeekCoder I encountered a strange behaviour.
When training the model earlier or later the loss goes to ...
- 1
0
votes
0
answers
14
views
How to derive the formula 13 in the Xavier Initialization paper
How to derive the formula 13 in the Xavier Initialization paper Understanding the difficulty of training deep feedforward neural networks from the formula 6?
- 791
1
vote
2
answers
143
views
Use of Gradient with respect to feature instead of model parameters
Generally, for any machine learning/deep learning system, we compute a loss, $L = l(x, \theta, y)$ which is a function of the input feature vector $x$ (after activation), model parameters $\theta$ (...
- 61
0
votes
1
answer
582
views
Gradient Boosting - Why pseudo-residuals?
I have some questions I don't really understand regarding the Gradient Boosting algorithm with Decision Trees:
Does the initial value matter as $\hat{y}$ or could you pick any, f.e between 0 and 1?
...
- 11
0
votes
1
answer
41
views
Does learning rate depend on input and output range?
I watched hours of videos on gradient descent and still feel pretty confused. Let's say I have a "model":
y = x * w
I use 2 as my target ...
- 103
0
votes
0
answers
2k
views
How to correctly create a PyTorch Tensor from a Pandas DataFrame?
I have loaded my data into a Pandas DataFrame, and performed some pre-processing, and then I need to convert it into a PyTorch Tensor for training as my features data.
Obviously, This new tensor do ...
- 101
1
vote
1
answer
56
views
How do we derive our loss function from the gradient objective?
I've been dwelling through RL theory and practice and one particular part I find hard to properly understand is the relation between the practical loss function and ...
- 111
0
votes
0
answers
149
views
calculating derivative of bias in backpropagation
Looking at the algorithm in wikipedia, we can implement backpropagation by calculating:
$$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$
(where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
- 101
0
votes
1
answer
649
views
Why would we add regularization loss to the gradient itself in an SVM?
I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM.
Relevant code:
...
- 125
1
vote
1
answer
24
views
Gradient Ascent and directional derivative
Suppose that you want to estimate a local maximum of the real function $f(x,y,z)$ with gradient ascent. Given a starting point $(x_0, y_0, z_0)$, the approach is to compute the gradient at this ...
- 115
1
vote
0
answers
55
views
How to interpret integrated gradients in an NLP toxic text classification use-case?
I am trying to understand how integrated gradients work in the NLP case.
Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
4
votes
1
answer
340
views
Differentiable approximation for counting negative values in array
I have an array of time of arrivals and I want to convert it to count data using pytorch in a differentiable way.
Example arrival times:
...
1
vote
0
answers
20
views
Central finite distance gradient simplified [closed]
I'm asked to compute central finite difference scheme (f(i+1)-f(i-1)) on an image. My attempt is something like:
...
1
vote
0
answers
79
views
Which Neural Network or Gradient Boosting framework is the simplest for Custom Loss Functions?
I need to implement a custom loss function.
The function is relatively simple:
$$-\sum \limits_{i=1}^m [O_{1,i} \cdot y_i-1] \ \cdot \ \operatorname{ReLu}(O_{1,i} \cdot \hat{y_i} - 1)$$
With $O$ being ...
- 199
3
votes
1
answer
7k
views
How batch normalization layer resolve the vanishing gradient problem?
According to this article:
https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484
The vanishing gradient problem occurs when using the sigmoid ...
- 527
0
votes
1
answer
67
views
Vanishing gradient problem
In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers?
2
votes
1
answer
1k
views
Why does my manual derivative of Layer Normalization imply no gradient flow?
I recently tried computing the derivative of the layer norm function (https://arxiv.org/abs/1607.06450), an essential component of transformers, but the result suggests that no gradient flows through ...
- 23
3
votes
3
answers
2k
views
Why a sign of gradient (plus or minus) is not enough for finding a steepest ascend?
Consider a simple 1-D function $y = x^2$ to find a maximum with the gradient ascent method.
If we start in point 3 on x-axis:
$$ \frac{\partial f}{\partial x} \...
2
votes
1
answer
531
views
Gradient passthough in PyTorch
I need to quantize the inputs, but the method (bucketize) I need to do so is indifferentiable. I can of course detach the tensor, but then I lose the flow of gradients to earlier weights. I guess ...
- 203
1
vote
1
answer
926
views
vanishing gradient and gradient zero
There is a well known problem vanishing gradient in BackPropagation training of ...
- 127
2
votes
1
answer
155
views
How to choose appropriate epsilon value while approximating gradients to check training?
While approximating gradients, using actual epsilon to shift the weights results in wildly big gradient approximations, as the "width" of the used approximation triangle is ...
- 145
0
votes
1
answer
1k
views
implementing forward and backward of a Linear model
I'm implementing the code of this abstraction.
The forward is easy and looks like that:
I don't understand the backward path and how it fit's the abstraction in the first image:
Why is db defined as ...
- 157
0
votes
2
answers
89
views
Intuitive explanation for representing gradient in higher dimensions
I do not understand how complex networks with many parameters/dimensions can be represented in a 3D space, and form a standard cost surface just like a simple network with, say, 2 parameters.
For ...
1
vote
1
answer
160
views
Can mini-batch gradient descent outperform batch gradient descent? [duplicate]
As I was reading and going through the second course of Andrew Ng's deep learning course, I came across a sentence that said,
With a well-turned mini-batch size, usually it outperforms either
...
- 153
2
votes
1
answer
1k
views
Tensorflow.Keras: How to get gradient for an output class w.r.t a given input?
I have implemented and trained a sequential model using tf.keras. Say I am given an input array of size 8X8 and an output [0,1,0,...(rest all 0)].
How to calculate the gradient of the input w.r.t to ...
1
vote
1
answer
128
views
CNN gradients with different magnitude
I have a CNN architecture with two cross entropy losses $\mathcal{L}_1$ and $\mathcal{L}_2$ summed in the total loss $\mathcal{L} = \mathcal{L}_1 + \mathcal{L}_2$. The task I want to solve is ...
- 117
0
votes
1
answer
180
views
when x is a vector, derivative of vector diag(f'(x)) is formal notation?
https://web.stanford.edu/class/cs224n/readings/gradient-notes.pdf
(4)
this note says this
$$
\frac{\partial \textbf{z}}{\partial \textbf{x}} = \text{diag}(f'(\textbf{x}))
$$
I know this means make ...
0
votes
1
answer
3k
views
Gradient of a function in Python
I've defined a function in this way:
def qfun(par):
return(par[0]+atan(par[3])*par[1]+atan(par[4])*par[2])
How can I obtain the gradient of this function ...
- 150
0
votes
1
answer
64
views
How can we get gradient with some other loss function apart from MSE?
In most of the gradient search examples, the update to weights are done by subtracting the derivative of MSE.
Can we have an example, where we did not use ...
1
vote
0
answers
43
views
Matlab Optimization. Meaning of warning: "The slope should be 2. It appears to be 1."
I'm using the manopt package to solve some optimization problems in matlab. The problem is of the form.
problem.cost = @(x) f(x)
problem.egrad = @(x) g(x)
After the problem definition, I check ...
- 111
3
votes
1
answer
99
views
Gradient Checking: MeanSquareError. Why huge epsilon improves discrepancy?
I am using custom C++ code, and coded a simple "Mean Squared Error" layer.
Temporarily using it for the 'classification task', not a simple regression. ...maybe this causes the issues?
I don't have ...
- 2,756
13
votes
4
answers
24k
views
How to prevent vanishing gradient or exploding gradient?
What's causing the vanishing gradient or exploding gradient, and what are the measures to be taken to prevent it?
- 169
2
votes
1
answer
396
views
Vanishing Gradient vs Exploding Gradient as Activation function?
ReLU is used as an activation function that serves two purposes:
Breaking linearity in DNN.
Helping in handling Vanishing Gradient problem.
For Exploding Gradient problem, we use Gradient Clipping ...
- 1,282
4
votes
3
answers
12k
views
Forward pass vs backward pass vs backpropagation
As mentioned in the question, I have some issues understanding what are the differences between those terms.
From what I have understood:
Forward pass: compute the output of the network given the ...
2
votes
1
answer
905
views
What does it mean for a method to be invariant to diagonal rescaling of the gradients?
In the paper which describes Adam: a method for stochastic optimization, the author states:
The method is straightforward to implement, is computationally
efficient, has little memory requirements, ...
- 121