Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [backpropagation]

Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
1 vote
0 answers
28 views

Backpropgation for a single parameter on a rather simple network

Given the following network: I'm asked to write the backpropagation process for the $b_3$ parameter, where the loss function is $L(y,z_3)=(z_3-y)^2$ I'm not supposed to calculate any of the weights ...
Aishgadol's user avatar
  • 111
0 votes
1 answer
36 views

My custom neural network is converging but keras model not

in most cases it is probably the other way round but... I have implemented a basic MLP neural network structure with backpropagation. My data is just a shifted quadratic function with 100 samples. I ...
tymsoncyferki's user avatar
0 votes
1 answer
17 views

Why no scale parameter for skip connection addition?

For a simple skip connection $y = x@w + x$, the gradient dy/dx will be $w+1$. $$\frac {\partial y}{\partial x} = w +1$$ Is +1 a bit too large and can it overpower $...
mon's user avatar
  • 721
1 vote
1 answer
46 views

CS 224N Back Propagation and Margin Loss in Neural Networks

I was going through Stanford CS 224 lecture notes on Back propagation. Page 5 states: We can see from the max-margin loss that: ∂J /∂s = − ∂J/∂s(c) = −1 I'm not sure I understand why this is the ...
Hormigas's user avatar
  • 113
0 votes
1 answer
77 views

Why not Back propagate through time in LSTM , similar to RNN

I'm trying to implement RNN and LSTM , many-to-many architecture. I reasoned myself why BPTT is necessary in RNNs and it makes sense. But what doesn't make sense to me is, most of resources I went ...
Amith Adiraju's user avatar
0 votes
0 answers
39 views

Multiplying by Diagonal Matrix On Top of Standard Linear Regression

In minimizing $$min_x||Ax-b||$$ where $A$ is overdetermined, one could use least squares method. However, if there is another diagonal matrix $d$ which has $k$ unique entries along the diagonal with $...
Trevor Arashiro's user avatar
0 votes
1 answer
38 views

How to prevent update a pretrained model if a model is optimized with backpropagation in Pytorch?

I use Pytorch exclusively to develop my model, and these are components in my model and how it works: A generator An encoder: a pretrained, and should not updated. A loss function. Input is passed to ...
Jesse's user avatar
  • 101
0 votes
0 answers
130 views

Runtime Error: one of the variables needed for gradient computation has been modified by an inplace operation:

I have the following code for a reinforcement learning using proximal policy optimization. It gives the following run time error. ...
heyula's user avatar
  • 37
0 votes
0 answers
50 views

Doubts on a custom loss function for regression problems

From what I read, I know we don't use log loss or cross entropy for regression problems. However, the entire logic behind binary cross entropy(say) is to firstly squeeze the y_hat between 0 and 1 (...
the_he_man's user avatar
1 vote
0 answers
19 views

Are "textbook backpropagation" still relevant?

The above backpropagation algorithm is taken from Shalev Shwartz and Ben-David's textbook: Understanding Machine Learning. This algorithm is described in the same way as the one in Mostafa's textbook, ...
Fraïssé's user avatar
  • 119
0 votes
0 answers
12 views

Is the semicolon (;) notation used to indicate operations are performed concurrently in backpropagation algorithm by Bengio?

I am trying to understand the backpropagation algorithm in a multi-layer perceptron environment. Algorithm 6.4 Backward computation for the deep neural network of algorithm 6.3, which uses, in ...
Revolucion for Monica's user avatar
0 votes
0 answers
33 views

Relu derivative value

I have a stupid question on the derivative of relu activation function. After the finding the difference of the true output $t_k$ and predicted output $a_k$, why is the value of the $d_{a3}$ \ $d_{z3}$...
Gunners 's user avatar
0 votes
1 answer
59 views

This simple python Feed forward Neural Network isn't learning. What am I doing wrong?

The backpropagation procedure is taken from the approach outlined in here. Here is the code, commented: ...
blundered_bishop's user avatar
0 votes
1 answer
142 views

How is the backward propagation is done in pytroch? When to use torch.no_grad, also when and where is the gradinte calcuated?

I have this training loop in pytorch. the loss_fn = nn.CrossEntropyLoss() and optim = torch.optim.Adam(net.parameters(), lr=lr) <...
Ahmed Gado's user avatar
0 votes
0 answers
17 views

Why does weight decay produce regularisation in Deep Neural Network?

Weight decay penalizes the model to have smaller weights but how does this help in regularisation? Any intuition on smaller weights => simpler model?
Sushil Khadka's user avatar
1 vote
0 answers
51 views

Confusion over taking gradients in Variational Autoencoder (VAE)

I am confused as to when to hold certain parameters constant in a VAE. I will explain with a concrete example. We can write $\operatorname{ELBO}(\phi, \theta) = \mathbb{E}_{q_{\phi}(z)}\left[\log \...
Joel's user avatar
  • 11
0 votes
1 answer
52 views

Backpropagation for each dimension of output in Pytorch

In PyTorch when we call loss.backward() it performs backpropagation for the sample (for stochastic case). Let’s consider my output is 50 dimensional. I have two loss components. First one is an array ...
Niloy Talukder's user avatar
2 votes
1 answer
371 views

Gradients of lower layers of NN when gradient of an upper layer is 0?

Say we have a neural network with an input layer, a hidden layer and an output layer. Say the gradients with respect to the weights and biases of the output layer are all 0. Then, by backpropagation ...
VJ123's user avatar
  • 147
0 votes
0 answers
47 views

Backpropagation and Gradient Descent: Questions on math behind it

I watched this video which goes over backpropagation calculus and read the Wikipedia page on it. This is my understanding of the equations for the algorithm. I have questions regarding the equations ...
notaorb's user avatar
  • 101
0 votes
1 answer
49 views

Problem for a math formula in Weight Uncertainty in Neural Network

I am studying the paper https://arxiv.org/pdf/1505.05424.pdf and there is a formula I don't get page 4: I don't understand how they obtain this formula. Moreover, with chain rule, I get $\frac{\...
Jack21's user avatar
  • 1
0 votes
1 answer
31 views

Neural Nets: Difference between activation and activation function, error on Wikipedia?

I'm reading the Wikipedia page on backpropagation and have some questions about the following equations: $$ \frac{d C}{d a^L}\cdot \frac{d a^L}{d z^L} \cdot \frac{d z^L}{d a^{L-1}} \cdot \frac{d a^{L-...
Nick's user avatar
  • 101
0 votes
0 answers
42 views

How to backpropagate transposed convolution with stride and padding

Please, help! I have deadlines and I do not have time to figure out the topic on my own. And now about the problem. I'm currently trying to figure out back propagation in transposed convolution. I ...
James's user avatar
  • 1
0 votes
0 answers
21 views

Machine Learning Geographical Data with NaNs

I have some features (physical properties related to geographical etc. of the Earth), I have a target that I'd like to predict. Sometimes this target is covered with clouds so I cannot see its actual ...
Socorro's user avatar
  • 111
0 votes
0 answers
29 views

How to backpropagate transposed convolution?

I'm currently learning Convolutional Neural Networks and am stuck on trying to figure out how to compute gradients in a layer that uses transposed convolution. Also, how do I calculate the gradients ...
Jakob's user avatar
  • 1
0 votes
0 answers
23 views

How does relu appears in first layer gradient of backpropagation?

I'm following Stanford's Natural language processing course in Coursera. I'm learning about "Continuous bag of words" model Where neural network with one relu(first layer) and one softmax(...
MathematicsBeginner's user avatar
0 votes
0 answers
106 views

Is there any reference about backpropagation of the Transformer's multi-head layer?

Is there any reference about backpropagation of the Transformer's multi-head layer or multi-head attention (MHA)? I have searched various journals but have not found one yet.
poglhar's user avatar
  • 25
2 votes
2 answers
5k views

What exactly is Gradient norm?

I found that there is no common resource and well defined definition for "Gradient norm", most search results are based on ML experts providing answers which involves gradient norm or papers ...
StudentV's user avatar
1 vote
1 answer
64 views

GAN Output Gradient Calculation

Loss function for discriminator, which needs to be maximized: -log(D(x)) + log(1-D(G(z))). Loss function for generator, which needs to be maximized: log(D(G(z))) What would the calculation of the loss ...
David's user avatar
  • 11
0 votes
1 answer
70 views

How exactly does the mini batch method work?

I mean let's say I have a mini batch, I take an example from it and for it I do the following: I do forward propagation. Using the output after forward propogation - I calculate the gradients of the ...
David's user avatar
  • 11
0 votes
0 answers
35 views

How to find the derivative of the hidden state of recurrent neural networks?

Recently I am reading the following paper (link) ...
user153245's user avatar
0 votes
3 answers
396 views

Regression model doesn't handle negative values

I'm trying to create a model that, given a feature $x_i$, predicts $y_i$ such that $y_i=ax^2_i+bx_i+c$ by using backpropagation. To do this, I'm using the ReLU activation function for each layer. The ...
Iya Lee's user avatar
  • 152
0 votes
1 answer
26 views

moments of weight vectors in Adam

When performing backpropagation with Adam algorithm, are the moment and the second moment of the weight vectors calculated also for the weights in hidden layers?
Iya Lee's user avatar
  • 152
0 votes
1 answer
37 views

Are some weight gradients equal?

I want to create a 3 layers neural network from scratch to perform linear regression. The first and the second layer have 2 neurons, and the last layer has one neuron. Feature vector x is divided into ...
Iya Lee's user avatar
  • 152
1 vote
0 answers
23 views

How to calculate gradient of MSE in backpropagation? [duplicate]

I want to implement a neural network from scratch to solve linear regression by using backpropagation. I don't understand how to compute the gradient of the MSE cost function with respect to each ...
Iya Lee's user avatar
  • 152
0 votes
1 answer
115 views

GAN Generator Backpropagation Gradient Shape Doesn't Match

In the TensorFlow example (https://www.tensorflow.org/tutorials/generative/dcgan#the_discriminator) the discriminator has a single output neuron (assume batch_size=1). Then over in the training loop ...
rkuang25's user avatar
0 votes
1 answer
143 views

Can a Simple Neural Network Predict a 0 or 1 Output by Looking Only at the Last Input?

I wrote a simple neural network that functions similarly to many of the C# examples I've seen online. It uses weights and biases and can be trained using backpropagation. It works well for ...
user1325179's user avatar
0 votes
1 answer
121 views

How does Back Propagation in a Neural Net Work?

I understand that, in a Neural Net, Back Propagation is used to update the model's weights and biases to lower loss, but how does this process actually work?
Connor's user avatar
  • 651
0 votes
0 answers
84 views

Why backpropagation is done in every epoch when loss is always scalar?

I understand the backpropagation algorithm that it calculates the derivate of loss with respect to all the parameters in the neural network. My question is this derivate is constant right because the ...
Jeet's user avatar
  • 101
0 votes
1 answer
80 views

How is loss calculated in truncated BPTT, for a many to one problem?

In many resources I refered to such as Justin Johnson's Lecture 12 on RNN, truncated BPTT is explained as the process of feedforward and backpropagate for smaller chunks of the sequence. These ...
butwhy's user avatar
  • 13
3 votes
1 answer
131 views

Why is it an advantage "that Markov chains are never needed" to obtain gradients?

In the original GAN (Generative Adversarial Network) paper, Generative adversarial networks by I. Goodfellow, J. Pouget-Abadie, M. Mirza et. al. they state an advantage of the GAN is "that Markov ...
p1unge's user avatar
  • 33
1 vote
1 answer
63 views

Is backpropagation applied every layer the same?

For example, I have layers that are pretrained. But while predicted, the loss is very high. But not because of pre-trained layers. Because of not pretrained layers. Will every layer be affected by ...
canP's user avatar
  • 101
0 votes
1 answer
319 views

Can MLP model sequential data?

When modeling sequential data, RNNs are introduced as an improvement of MLP as they can model the time dependency between the inputs. It is said that feeding the last N data points in the sequence to ...
Lukas Petersson's user avatar
0 votes
1 answer
146 views

How can I make model unlearn? reverse backpropagation?

I stumbled upon a highly dimensional minimum that I can't seem to reproduce no matter how many hundreds of models I train. The problem is that I went a few epochs too far and overfit on the training ...
Kermit's user avatar
  • 529
0 votes
1 answer
85 views

Back propagation matrix shape error using Python

I wanna implement the back-propagation algorithm in python with the next code ...
Al.Vioky's user avatar
0 votes
0 answers
144 views

calculating derivative of bias in backpropagation

Looking at the algorithm in wikipedia, we can implement backpropagation by calculating: $$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$ (where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
Ariel Yael's user avatar
0 votes
1 answer
15 views

evaluation of gradient for the subset of parameters using backpropagation

Consider simple feed-forward neural network with few layers. I would like to evaluate only gradients of one particular layer, denoted by X. This should be performed repetitively, while parameters of ...
student1's user avatar
0 votes
0 answers
17 views

Backtracking filter coefficients of Convolutional Neural Networks

I'm starting to learn how convolutional neural networks work, and I have a question regarding the filters. Apparently, these are randomly generated when the model is generated, and then as the data is ...
Juan Cruz Carrau's user avatar
0 votes
0 answers
82 views

Is Loss value (e.g., MSE loss) used in the calculation for parameter update when doing gradient descent?

My question is really simple. I know the theory behind gradient descent and parameter updates, what I really haven't found clarity on is that is the loss value (e.g., MSE value) used, i.e., multiplied ...
AZ123's user avatar
  • 11
1 vote
0 answers
244 views

How to plot the computational graph and derive the update procedure of parameters using the backpropagation algorithm?

Please help me to solve this problem without a code (ps: this is a written problem): Given the following loss function, please plot the computational graph, and derive the update procedure of ...
Nezuko's user avatar
  • 21
1 vote
0 answers
59 views

How to compute backpropagation gradient according chain rule for using vector/matrix differential?

I have some problems for computing derivative for sum of squares error in backprop neural network. For example, we have a neural network as in picture. For drawing simplicity, i've dropped the sample ...
Grigogiy Reznichenko's user avatar

1
2 3 4 5
7