The tag has no usage guidance.

39 questions
Filter by
Sorted by
Tagged with
1 vote
18 views

How do we derive our loss function from the gradient objective?

I've been dwelling through RL theory and practice and one particular part I find hard to properly understand is the relation between the practical loss function and ...
5 views

Defining a Python operation with a gradient in TensorFlow 2

I am trying to understand how I can define my own function in python, and then define its derivative so that it can be used in a GradientTape scope. I have found this answer which is for TensorFlow 1, ...
13 views

calculating derivative of bias in backpropagation

Looking at the algorithm in wikipedia, we can implement backpropagation by calculating: $$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$ (where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
• 101
5 views

How to calculate the expression for the gradient of softmax + cross entropy with respect to weights?

I'm learning cs231n on my own. The Softmax classifier has the following loss function: to make this clear: $L_i$ is the loss for a particular training input $f_j$ is the $j$th element of the vector, ...
• 115
25 views

Why would we add regularization loss to the gradient itself in an SVM?

I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM. Relevant code: ...
• 115
3 views

Suppose that you want to estimate a local maximum of the real function $f(x,y,z)$ with gradient ascent. Given a starting point $(x_0, y_0, z_0)$, the approach is to compute the gradient at this ...
• 89
49 views

How does the margin constant (alpha) in the triplet loss affect the training process when it is a constant?

How does the margin constant in the triplet loss formula affect the gradient calculation when its derivative will be zero?
• 101
9 views

Gradient and loss calculation localization in Vision Transformers

Hi all I am resorting to you to figure out where the gradient and the loss for q,k,v weights update happens in Vision Transformers. I suspect it is the MLP/FF bit of the architecture but I am not ...
16 views

Backpropagation in NN

During backward pass, which gradients are kept and which gradients are discarded? Why are some gradients discarded? I know that forward pass is computing the output of the network given the inputs and ...
12 views

On page 58 of the second edition of Deep Learning with Python, Chollet is illustrating an example of a forward and backward pass of a computation graph. The computation graph is given by: $$x\to w\... • 101 1 vote 0 answers 7 views How to interpret integrated gradients in an NLP toxic text classification use-case? I am trying to understand how integrated gradients work in the NLP case. Let F: \mathbb{R}^{n} \rightarrow[0,1] a function representing a neural network, x \in \mathbb{R}^{n} an input and x' \in ... 0 votes 0 answers 14 views Is there a difference between AutoGrad and explicit derivatives (gradient)? Will there be some differences between applying AutoGrad on the loss function (using a python library) and applying explicit gradient (the gradient from the paper or the update rule)? For example: ... 0 votes 0 answers 20 views How to manually calculate the gradient that will propagate back over the network using the REINFORCE algorithm? I am trying to implement deep reinforcement policy gradient REINFORCE in C++ and for my case there is no "autograd" method like in pytorch so I have to manually calculate the gradient. Let´s ... 0 votes 0 answers 11 views Analytical gradients from tf.gradients don't match approximate gradients I have a trained neural network (NN) with independent inputs x1, x2.. xn and a scalar output y. Input ... 0 votes 0 answers 7 views Propagating -infs in pytorch and outliers in general I am using a loss which requires sampling from probability distributions to do monte carlo integration with. Sometimes legitimate training data can throw -inf/NaN. ... 2 votes 1 answer 143 views Differentiable approximation for counting negative values in array I have an array of time of arrivals and I want to convert it to count data using pytorch in a differentiable way. Example arrival times: ... 0 votes 0 answers 20 views Gradients are becoming None in PyTorch ... • 209 0 votes 0 answers 19 views Why is the signed gradient of the image used for adversarial examples [duplicate] In this paper, the gradient of the loss w.r.t. to the image is computed, but its sign is used. Why is using the sign-method better? • 141 1 vote 0 answers 12 views Central finite distance gradient simplified [closed] I'm asked to compute central finite difference scheme (f(i+1)-f(i-1)) on an image. My attempt is something like: ... 1 vote 0 answers 58 views Which Neural Network or Gradient Boosting framework is the simplest for Custom Loss Functions? I need to implement a custom loss function. The function is relatively simple:$$-\sum \limits_{i=1}^m [O_{1,i} \cdot y_i-1] \ \cdot \ \operatorname{ReLu}(O_{1,i} \cdot \hat{y_i} - 1)$$With O being ... • 179 2 votes 1 answer 2k views How batch normalization layer resolve the vanishing gradient problem? According to this article: https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484 The vanishing gradient problem occurs when using the sigmoid ... • 317 0 votes 1 answer 33 views Vanishing gradient problem In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers? 1 vote 1 answer 201 views Why does my manual derivative of Layer Normalization imply no gradient flow? I recently tried computing the derivative of the layer norm function (https://arxiv.org/abs/1607.06450), an essential component of transformers, but the result suggests that no gradient flows through ... • 13 3 votes 3 answers 1k views Why a sign of gradient (plus or minus) is not enough for finding a steepest ascend? Consider a simple 1-D function y = x^2 to find a maximum with the gradient ascent method. If we start in point 3 on x-axis:$$ \frac{\partial f}{\partial x} \...
85 views

I need to quantize the inputs, but the method (bucketize) I need to do so is indifferentiable. I can of course detach the tensor, but then I lose the flow of gradients to earlier weights. I guess ...
• 203
1 vote
454 views

There is a well known problem vanishing gradient in BackPropagation training of ...
• 127
1 vote
37 views

How to choose appropriate epsilon value while approximating gradients to check training?

While approximating gradients, using actual epsilon to shift the weights results in wildly big gradient approximations, as the "width" of the used approximation triangle is ...
• 135
109 views

implementing forward and backward of a Linear model

I'm implementing the code of this abstraction. The forward is easy and looks like that: I don't understand the backward path and how it fit's the abstraction in the first image: Why is db defined as ...
• 157
48 views

Intuitive explanation for representing gradient in higher dimensions

I do not understand how complex networks with many parameters/dimensions can be represented in a 3D space, and form a standard cost surface just like a simple network with, say, 2 parameters. For ...
1 vote
89 views

As I was reading and going through the second course of Andrew Ng's deep learning course, I came across a sentence that said, With a well-turned mini-batch size, usually it outperforms either ...
956 views

Tensorflow.Keras: How to get gradient for an output class w.r.t a given input?

I have implemented and trained a sequential model using tf.keras. Say I am given an input array of size 8X8 and an output [0,1,0,...(rest all 0)]. How to calculate the gradient of the input w.r.t to ...
1 vote
75 views

I have a CNN architecture with two cross entropy losses $\mathcal{L}_1$ and $\mathcal{L}_2$ summed in the total loss $\mathcal{L} = \mathcal{L}_1 + \mathcal{L}_2$. The task I want to solve is ...
• 117
74 views

when x is a vector, derivative of vector diag(f'(x)) is formal notation?

https://web.stanford.edu/class/cs224n/readings/gradient-notes.pdf (4) this note says this $$\frac{\partial \textbf{z}}{\partial \textbf{x}} = \text{diag}(f'(\textbf{x}))$$ I know this means make ...
2k views

Gradient of a function in Python

I've defined a function in this way: def qfun(par): return(par[0]+atan(par[3])*par[1]+atan(par[4])*par[2]) How can I obtain the gradient of this function ...
• 150
50 views

How can we get gradient with some other loss function apart from MSE?

In most of the gradient search examples, the update to weights are done by subtracting the derivative of MSE. Can we have an example, where we did not use ...
1 vote
23 views

Matlab Optimization. Meaning of warning: "The slope should be 2. It appears to be 1."

I'm using the manopt package to solve some optimization problems in matlab. The problem is of the form. problem.cost = @(x) f(x) problem.egrad = @(x) g(x) After the problem definition, I check ...
• 111
64 views

Gradient Checking: MeanSquareError. Why huge epsilon improves discrepancy?

I am using custom C++ code, and coded a simple "Mean Squared Error" layer. Temporarily using it for the 'classification task', not a simple regression. ...maybe this causes the issues? I don't have ...
• 2,586
141 views