Questions tagged [gradient]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
31 views

Problem of constant shift in prediction for neural network regression model with gradient-domain loss function

I'm training a regression model using neural network which is trained on MSE of both output and spatial gradient of output. With some simplification, the model is: $$ y = f(\mathbf{x};\theta) $$ where ...
mjhong's user avatar
  • 1
0 votes
0 answers
20 views

Understanding gradient of skip gram

I am trying to understand gradient calculation for skip gram with softmax output and cross entropy loss. I am referring these articles: 1, 2, 3. The all calculate the error as follows: $$E=-\sum_{c=1}...
RajS's user avatar
  • 103
0 votes
0 answers
64 views

How does tensorflow keras lambda layer pass gradients?

Tensorflow / Keras has a Lambda layer which aids in working with arbitrary expressions as a Layer object. In my understanding a layer should be differentiable so that gradients can propagate through ...
Eka's user avatar
  • 291
0 votes
0 answers
2k views

How to correctly create a PyTorch Tensor from a Pandas DataFrame?

I have loaded my data into a Pandas DataFrame, and performed some pre-processing, and then I need to convert it into a PyTorch Tensor for training as my features data. Obviously, This new tensor do ...
EvilRoach's user avatar
  • 101
1 vote
1 answer
27 views

How do we derive our loss function from the gradient objective?

I've been dwelling through RL theory and practice and one particular part I find hard to properly understand is the relation between the practical loss function and ...
Alex Ramalho's user avatar
0 votes
0 answers
117 views

calculating derivative of bias in backpropagation

Looking at the algorithm in wikipedia, we can implement backpropagation by calculating: $$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$ (where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
Ariel Yael's user avatar
0 votes
1 answer
268 views

Why would we add regularization loss to the gradient itself in an SVM?

I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM. Relevant code: ...
Foobar's user avatar
  • 115
1 vote
1 answer
10 views

Gradient Ascent and directional derivative

Suppose that you want to estimate a local maximum of the real function $f(x,y,z)$ with gradient ascent. Given a starting point $(x_0, y_0, z_0)$, the approach is to compute the gradient at this ...
Enk9456's user avatar
  • 83
1 vote
0 answers
30 views

How to interpret integrated gradients in an NLP toxic text classification use-case?

I am trying to understand how integrated gradients work in the NLP case. Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
Revolucion for Monica's user avatar
3 votes
1 answer
261 views

Differentiable approximation for counting negative values in array

I have an array of time of arrivals and I want to convert it to count data using pytorch in a differentiable way. Example arrival times: ...
iRestMyCaseYourHonor's user avatar
1 vote
0 answers
19 views

Central finite distance gradient simplified [closed]

I'm asked to compute central finite difference scheme (f(i+1)-f(i-1)) on an image. My attempt is something like: ...
Anđela Todorović's user avatar
1 vote
0 answers
72 views

Which Neural Network or Gradient Boosting framework is the simplest for Custom Loss Functions?

I need to implement a custom loss function. The function is relatively simple: $$-\sum \limits_{i=1}^m [O_{1,i} \cdot y_i-1] \ \cdot \ \operatorname{ReLu}(O_{1,i} \cdot \hat{y_i} - 1)$$ With $O$ being ...
Borut Flis's user avatar
3 votes
1 answer
4k views

How batch normalization layer resolve the vanishing gradient problem?

According to this article: https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484 The vanishing gradient problem occurs when using the sigmoid ...
user3668129's user avatar
0 votes
1 answer
51 views

Vanishing gradient problem

In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers?
Domenico Bagnato's user avatar
1 vote
1 answer
627 views

Why does my manual derivative of Layer Normalization imply no gradient flow?

I recently tried computing the derivative of the layer norm function (https://arxiv.org/abs/1607.06450), an essential component of transformers, but the result suggests that no gradient flows through ...
Alex's user avatar
  • 13
3 votes
3 answers
2k views

Why a sign of gradient (plus or minus) is not enough for finding a steepest ascend?

Consider a simple 1-D function $y = x^2$ to find a maximum with the gradient ascent method. If we start in point 3 on x-axis: $$ \frac{\partial f}{\partial x} \...
Kenenbek Arzymatov's user avatar
2 votes
1 answer
268 views

Gradient passthough in PyTorch

I need to quantize the inputs, but the method (bucketize) I need to do so is indifferentiable. I can of course detach the tensor, but then I lose the flow of gradients to earlier weights. I guess ...
user3023715's user avatar
1 vote
1 answer
767 views

vanishing gradient and gradient zero

There is a well known problem vanishing gradient in BackPropagation training of ...
user6703592's user avatar
2 votes
1 answer
69 views

How to choose appropriate epsilon value while approximating gradients to check training?

While approximating gradients, using actual epsilon to shift the weights results in wildly big gradient approximations, as the "width" of the used approximation triangle is ...
Dávid Tóth's user avatar
0 votes
1 answer
468 views

implementing forward and backward of a Linear model

I'm implementing the code of this abstraction. The forward is easy and looks like that: I don't understand the backward path and how it fit's the abstraction in the first image: Why is db defined as ...
Ilya.K.'s user avatar
  • 157
0 votes
2 answers
70 views

Intuitive explanation for representing gradient in higher dimensions

I do not understand how complex networks with many parameters/dimensions can be represented in a 3D space, and form a standard cost surface just like a simple network with, say, 2 parameters. For ...
forhayley 's user avatar
1 vote
1 answer
121 views

Can mini-batch gradient descent outperform batch gradient descent? [duplicate]

As I was reading and going through the second course of Andrew Ng's deep learning course, I came across a sentence that said, With a well-turned mini-batch size, usually it outperforms either ...
mitra mirshafiee's user avatar
2 votes
1 answer
1k views

Tensorflow.Keras: How to get gradient for an output class w.r.t a given input?

I have implemented and trained a sequential model using tf.keras. Say I am given an input array of size 8X8 and an output [0,1,0,...(rest all 0)]. How to calculate the gradient of the input w.r.t to ...
samarendra chandan bindu Dash's user avatar
1 vote
1 answer
89 views

CNN gradients with different magnitude

I have a CNN architecture with two cross entropy losses $\mathcal{L}_1$ and $\mathcal{L}_2$ summed in the total loss $\mathcal{L} = \mathcal{L}_1 + \mathcal{L}_2$. The task I want to solve is ...
aretor's user avatar
  • 117
0 votes
1 answer
119 views

when x is a vector, derivative of vector diag(f'(x)) is formal notation?

https://web.stanford.edu/class/cs224n/readings/gradient-notes.pdf (4) this note says this $$ \frac{\partial \textbf{z}}{\partial \textbf{x}} = \text{diag}(f'(\textbf{x})) $$ I know this means make ...
NeverneverNever's user avatar
0 votes
1 answer
2k views

Gradient of a function in Python

I've defined a function in this way: def qfun(par): return(par[0]+atan(par[3])*par[1]+atan(par[4])*par[2]) How can I obtain the gradient of this function ...
unter_983's user avatar
  • 150
0 votes
1 answer
62 views

How can we get gradient with some other loss function apart from MSE?

In most of the gradient search examples, the update to weights are done by subtracting the derivative of MSE. Can we have an example, where we did not use ...
The DataScience's user avatar
1 vote
0 answers
27 views

Matlab Optimization. Meaning of warning: "The slope should be 2. It appears to be 1."

I'm using the manopt package to solve some optimization problems in matlab. The problem is of the form. problem.cost = @(x) f(x) problem.egrad = @(x) g(x) After the problem definition, I check ...
Springberg's user avatar
3 votes
1 answer
81 views

Gradient Checking: MeanSquareError. Why huge epsilon improves discrepancy?

I am using custom C++ code, and coded a simple "Mean Squared Error" layer. Temporarily using it for the 'classification task', not a simple regression. ...maybe this causes the issues? I don't have ...
Kari's user avatar
  • 2,686
2 votes
1 answer
265 views

Vanishing Gradient vs Exploding Gradient as Activation function?

ReLU is used as an activation function that serves two purposes: Breaking linearity in DNN. Helping in handling Vanishing Gradient problem. For Exploding Gradient problem, we use Gradient Clipping ...
vipin bansal's user avatar
  • 1,252
2 votes
1 answer
825 views

What does it mean for a method to be invariant to diagonal rescaling of the gradients?

In the paper which describes Adam: a method for stochastic optimization, the author states: The method is straightforward to implement, is computationally efficient, has little memory requirements, ...
Erudite's user avatar
  • 121