# Questions tagged [gradient]

The tag has no usage guidance.

25 questions
Filter by
Sorted by
Tagged with
0answers
5 views

### Gradient calculation of the pre-trained model

I have pre-trained tensorflow model in a graph format. I have not used tf.gradient on the graph structure. In such cases, is there any way to calculate the gradient of some tensors operation?
0answers
11 views

### Gradients vanishing despite using Kaiming initialization

I was implementing a conv block in pytorch with activation function(prelu). I used Kaiming initilization to initialize all my weights and set all the bias to zero. However as I tested these blocks (by ...
0answers
22 views

### Gradient calculation analysis

I'm using VGG16 pretrained architecture for classification and visualization of result using Guided Backpropagation technique. I have used tensorflow code for calaulating the input gradientSharing ...
1answer
18 views

### Vanishing gradient problem

In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers?
1answer
31 views

### Why does my manual derivative of Layer Normalization imply no gradient flow?

I recently tried computing the derivative of the layer norm function (https://arxiv.org/abs/1607.06450), an essential component of transformers, but the result suggests that no gradient flows through ...
0answers
11 views

### How can my loss be stable while the gradient keeps growing?

I have been working on an Offline/Batch Reinforcement Learning problem where I am using a BCQ-DDQN model as a Q-table. The model input is a state of 8 dimensions, and the output is a vector of Q-...
3answers
614 views

1answer
213 views

### Gradient of a function in Python

I've defined a function in this way: def qfun(par): return(par+atan(par)*par+atan(par)*par) How can I obtain the gradient of this function ...
1answer
24 views

### How can we get gradient with some other loss function apart from MSE?

In most of the gradient search examples, the update to weights are done by subtracting the derivative of MSE. Can we have an example, where we did not use ...
0answers
18 views

### Matlab Optimization. Meaning of warning: “The slope should be 2. It appears to be 1.”

I'm using the manopt package to solve some optimization problems in matlab. The problem is of the form. problem.cost = @(x) f(x) problem.egrad = @(x) g(x) After the problem definition, I check ...
1answer
47 views

### Gradient Checking: MeanSquareError. Why huge epsilon improves discrepancy?

I am using custom C++ code, and coded a simple "Mean Squared Error" layer. Temporarily using it for the 'classification task', not a simple regression. ...maybe this causes the issues? I don't have ...
1answer
72 views

### Vanishing Gradient vs Exploding Gradient as Activation function?

ReLU is used as an activation function that serves two purposes: Breaking linearity in DNN. Helping in handling Vanishing Gradient problem. For Exploding Gradient problem, we use Gradient Clipping ...
1answer
531 views

### What does it mean for a method to be invariant to diagonal rescaling of the gradients?

In the paper which describes Adam: a method for stochastic optimization, the author states: The method is straightforward to implement, is computationally efficient, has little memory requirements, ...