Questions tagged [backpropagation]

Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
0 votes
0 answers
38 views

Doubts on a custom loss function for regression problems

From what I read, I know we don't use log loss or cross entropy for regression problems. However, the entire logic behind binary cross entropy(say) is to firstly squeeze the y_hat between 0 and 1 (...
the_he_man's user avatar
1 vote
0 answers
17 views

Are "textbook backpropagation" still relevant?

The above backpropagation algorithm is taken from Shalev Shwartz and Ben-David's textbook: Understanding Machine Learning. This algorithm is described in the same way as the one in Mostafa's textbook, ...
Fraïssé's user avatar
  • 119
0 votes
0 answers
12 views

Is the semicolon (;) notation used to indicate operations are performed concurrently in backpropagation algorithm by Bengio?

I am trying to understand the backpropagation algorithm in a multi-layer perceptron environment. Algorithm 6.4 Backward computation for the deep neural network of algorithm 6.3, which uses, in ...
Revolucion for Monica's user avatar
0 votes
0 answers
19 views

Relu derivative value

I have a stupid question on the derivative of relu activation function. After the finding the difference of the true output $t_k$ and predicted output $a_k$, why is the value of the $d_{a3}$ \ $d_{z3}$...
Gunners 's user avatar
0 votes
1 answer
48 views

This simple python Feed forward Neural Network isn't learning. What am I doing wrong?

The backpropagation procedure is taken from the approach outlined in here. Here is the code, commented: ...
blundered_bishop's user avatar
0 votes
1 answer
44 views

How is the backward propagation is done in pytroch? When to use torch.no_grad, also when and where is the gradinte calcuated?

I have this training loop in pytorch. the loss_fn = nn.CrossEntropyLoss() and optim = torch.optim.Adam(net.parameters(), lr=lr) <...
Ahmed Gado's user avatar
0 votes
0 answers
11 views

Why does weight decay produce regularisation in Deep Neural Network?

Weight decay penalizes the model to have smaller weights but how does this help in regularisation? Any intuition on smaller weights => simpler model?
Sushil Khadka's user avatar
1 vote
0 answers
28 views

Confusion over taking gradients in Variational Autoencoder (VAE)

I am confused as to when to hold certain parameters constant in a VAE. I will explain with a concrete example. We can write $\operatorname{ELBO}(\phi, \theta) = \mathbb{E}_{q_{\phi}(z)}\left[\log \...
Joel's user avatar
  • 11
0 votes
1 answer
29 views

Backpropagation for each dimension of output in Pytorch

In PyTorch when we call loss.backward() it performs backpropagation for the sample (for stochastic case). Let’s consider my output is 50 dimensional. I have two loss components. First one is an array ...
Niloy Talukder's user avatar
2 votes
1 answer
295 views

Gradients of lower layers of NN when gradient of an upper layer is 0?

Say we have a neural network with an input layer, a hidden layer and an output layer. Say the gradients with respect to the weights and biases of the output layer are all 0. Then, by backpropagation ...
VJ123's user avatar
  • 147
0 votes
0 answers
44 views

Backpropagation and Gradient Descent: Questions on math behind it

I watched this video which goes over backpropagation calculus and read the Wikipedia page on it. This is my understanding of the equations for the algorithm. I have questions regarding the equations ...
notaorb's user avatar
  • 101
0 votes
1 answer
32 views

Problem for a math formula in Weight Uncertainty in Neural Network

I am studying the paper https://arxiv.org/pdf/1505.05424.pdf and there is a formula I don't get page 4: I don't understand how they obtain this formula. Moreover, with chain rule, I get $\frac{\...
Jack21's user avatar
  • 1
0 votes
1 answer
28 views

Neural Nets: Difference between activation and activation function, error on Wikipedia?

I'm reading the Wikipedia page on backpropagation and have some questions about the following equations: $$ \frac{d C}{d a^L}\cdot \frac{d a^L}{d z^L} \cdot \frac{d z^L}{d a^{L-1}} \cdot \frac{d a^{L-...
Nick's user avatar
  • 101
0 votes
0 answers
21 views

How to backpropagate transposed convolution with stride and padding

Please, help! I have deadlines and I do not have time to figure out the topic on my own. And now about the problem. I'm currently trying to figure out back propagation in transposed convolution. I ...
James's user avatar
  • 1
0 votes
0 answers
19 views

Machine Learning Geographical Data with NaNs

I have some features (physical properties related to geographical etc. of the Earth), I have a target that I'd like to predict. Sometimes this target is covered with clouds so I cannot see its actual ...
Socorro's user avatar
  • 91
0 votes
0 answers
26 views

How to backpropagate transposed convolution?

I'm currently learning Convolutional Neural Networks and am stuck on trying to figure out how to compute gradients in a layer that uses transposed convolution. Also, how do I calculate the gradients ...
Jakob's user avatar
  • 1
0 votes
0 answers
23 views

How does relu appears in first layer gradient of backpropagation?

I'm following Stanford's Natural language processing course in Coursera. I'm learning about "Continuous bag of words" model Where neural network with one relu(first layer) and one softmax(...
MathematicsBeginner's user avatar
0 votes
0 answers
67 views

Is there any reference about backpropagation of the Transformer's multi-head layer?

Is there any reference about backpropagation of the Transformer's multi-head layer or multi-head attention (MHA)? I have searched various journals but have not found one yet.
poglhar's user avatar
  • 13
1 vote
2 answers
2k views

What exactly is Gradient norm?

I found that there is no common resource and well defined definition for "Gradient norm", most search results are based on ML experts providing answers which involves gradient norm or papers ...
StudentV's user avatar
1 vote
1 answer
31 views

GAN Output Gradient Calculation

Loss function for discriminator, which needs to be maximized: -log(D(x)) + log(1-D(G(z))). Loss function for generator, which needs to be maximized: log(D(G(z))) What would the calculation of the loss ...
David's user avatar
  • 11
0 votes
1 answer
51 views

How exactly does the mini batch method work?

I mean let's say I have a mini batch, I take an example from it and for it I do the following: I do forward propagation. Using the output after forward propogation - I calculate the gradients of the ...
David's user avatar
  • 11
0 votes
0 answers
28 views

How to find the derivative of the hidden state of recurrent neural networks?

Recently I am reading the following paper (link) ...
user153245's user avatar
0 votes
0 answers
14 views

forward or reverse accumulation DL frameworks

Automatic differentiation can be accomplished using forward or reverse accumulation. Quoting Wikipedia : which mode is used in DL frameworks is used for implementation and why? Does it have any ...
ujjwalnur's user avatar
0 votes
0 answers
32 views

Backpropagation of position-wise feedforward neural network

I have read a paper entitled "Attention is all you need" by Vaswani et al. (2017). This paper use the so-called position-wise feedforward neural network, where the input of this network is a ...
poglhar's user avatar
  • 13
0 votes
0 answers
16 views

Is It Fair to Simplify Gradient Descent as Derivative of Loss Function * Derivative of Activation Function * Derivative of Neuron?

Recently I've been trying to think of an easier way to wrap my head around gradient descent rather than just manually derive loss function with respect to weights (and chain and yada yada yada). I was ...
Hay's user avatar
  • 1
0 votes
3 answers
130 views

Regression model doesn't handle negative values

I'm trying to create a model that, given a feature $x_i$, predicts $y_i$ such that $y_i=ax^2_i+bx_i+c$ by using backpropagation. To do this, I'm using the ReLU activation function for each layer. The ...
Iya Lee's user avatar
  • 152
0 votes
1 answer
16 views

moments of weight vectors in Adam

When performing backpropagation with Adam algorithm, are the moment and the second moment of the weight vectors calculated also for the weights in hidden layers?
Iya Lee's user avatar
  • 152
0 votes
1 answer
36 views

Are some weight gradients equal?

I want to create a 3 layers neural network from scratch to perform linear regression. The first and the second layer have 2 neurons, and the last layer has one neuron. Feature vector x is divided into ...
Iya Lee's user avatar
  • 152
1 vote
0 answers
23 views

How to calculate gradient of MSE in backpropagation? [duplicate]

I want to implement a neural network from scratch to solve linear regression by using backpropagation. I don't understand how to compute the gradient of the MSE cost function with respect to each ...
Iya Lee's user avatar
  • 152
0 votes
0 answers
53 views

Bugs in the backpropagation algorithm in Python

I've been trying to create a simple Neural Network from scratch with a backpropagation algorithm to predict the next number based on 3 previous numbers. But for some reasons, MSE(Mean Squared Error) ...
Mr. White's user avatar
0 votes
0 answers
24 views

Back Propagation on arbitrary depth network with ReLu

I am implementing a neural network of arbitrary depth with an arbitrary number of nodes on each depth. My forwards propagation thus looks like this (For 2 hidden layers) ...
An Assembler's user avatar
0 votes
0 answers
65 views

How do i calculate 1 iteration of the backpropagation algorithm on this exercise

Hello, i'm currently learning Neural Networks, so have a lot of doubts on how do i calculate the weights after one iteration. The step function for neuron 1 is:y1 = step(x1 + x2 -1.5) The step ...
Tomás Oom's user avatar
0 votes
0 answers
41 views

I am creating an multilayer LSTM model from scratch and everything seems to be mathematically correct however the model refuses to learn

I am creating the LSTM with just numpy and plotting the loss with pyplot. I have checked the derivatives again and again however have not found a mistake. The entire code with the main function can be ...
Lanttu1243's user avatar
0 votes
0 answers
19 views

Function to watch out to avoid infinity during back propagation

Math functions can cause infinity during back propagations, e.g. derivative of sqrt(x) can be infinitive when x==0. $$\frac {d}{...
mon's user avatar
  • 585
0 votes
0 answers
18 views

How to derive expression for gradient in BPPT

I have the following problem: I am trying to derive final expressions for error gradients in a simple recurrent neural network (Backpropagation through Time, BPPT). The parameters and state update ...
doktormatte's user avatar
0 votes
1 answer
95 views

GAN Generator Backpropagation Gradient Shape Doesn't Match

In the TensorFlow example (https://www.tensorflow.org/tutorials/generative/dcgan#the_discriminator) the discriminator has a single output neuron (assume batch_size=1). Then over in the training loop ...
rkuang25's user avatar
0 votes
1 answer
98 views

Can a Simple Neural Network Predict a 0 or 1 Output by Looking Only at the Last Input?

I wrote a simple neural network that functions similarly to many of the C# examples I've seen online. It uses weights and biases and can be trained using backpropagation. It works well for ...
user1325179's user avatar
0 votes
1 answer
83 views

How does Back Propagation in a Neural Net Work?

I understand that, in a Neural Net, Back Propagation is used to update the model's weights and biases to lower loss, but how does this process actually work?
Connor's user avatar
  • 617
0 votes
0 answers
19 views

How does backpropagation through accuracy work?

I'm using a specific constraint on my predicted logits and adding it to the loss. In a nutshell, this constraint tries to minimize cross-overlap between the channels of my predictions. I'm using ...
debrises's user avatar
0 votes
0 answers
65 views

Why backpropagation is done in every epoch when loss is always scalar?

I understand the backpropagation algorithm that it calculates the derivate of loss with respect to all the parameters in the neural network. My question is this derivate is constant right because the ...
Jeet's user avatar
  • 101
0 votes
1 answer
53 views

How is loss calculated in truncated BPTT, for a many to one problem?

In many resources I refered to such as Justin Johnson's Lecture 12 on RNN, truncated BPTT is explained as the process of feedforward and backpropagate for smaller chunks of the sequence. These ...
butwhy's user avatar
  • 13
3 votes
1 answer
101 views

Why is it an advantage "that Markov chains are never needed" to obtain gradients?

In the original GAN (Generative Adversarial Network) paper, Generative adversarial networks by I. Goodfellow, J. Pouget-Abadie, M. Mirza et. al. they state an advantage of the GAN is "that Markov ...
p1unge's user avatar
  • 33
1 vote
1 answer
59 views

Is backpropagation applied every layer the same?

For example, I have layers that are pretrained. But while predicted, the loss is very high. But not because of pre-trained layers. Because of not pretrained layers. Will every layer be affected by ...
canP's user avatar
  • 91
0 votes
1 answer
215 views

Can MLP model sequential data?

When modeling sequential data, RNNs are introduced as an improvement of MLP as they can model the time dependency between the inputs. It is said that feeding the last N data points in the sequence to ...
Lukas Petersson's user avatar
0 votes
1 answer
103 views

How can I make model unlearn? reverse backpropagation?

I stumbled upon a highly dimensional minimum that I can't seem to reproduce no matter how many hundreds of models I train. The problem is that I went a few epochs too far and overfit on the training ...
Kermit's user avatar
  • 519
0 votes
1 answer
55 views

Back propagation matrix shape error using Python

I wanna implement the back-propagation algorithm in python with the next code ...
Al.Vioky's user avatar
0 votes
0 answers
132 views

calculating derivative of bias in backpropagation

Looking at the algorithm in wikipedia, we can implement backpropagation by calculating: $$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$ (where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
Ariel Yael's user avatar
0 votes
1 answer
14 views

evaluation of gradient for the subset of parameters using backpropagation

Consider simple feed-forward neural network with few layers. I would like to evaluate only gradients of one particular layer, denoted by X. This should be performed repetitively, while parameters of ...
student1's user avatar
0 votes
0 answers
13 views

Backtracking filter coefficients of Convolutional Neural Networks

I'm starting to learn how convolutional neural networks work, and I have a question regarding the filters. Apparently, these are randomly generated when the model is generated, and then as the data is ...
Juan Cruz Carrau's user avatar
0 votes
0 answers
79 views

Is Loss value (e.g., MSE loss) used in the calculation for parameter update when doing gradient descent?

My question is really simple. I know the theory behind gradient descent and parameter updates, what I really haven't found clarity on is that is the loss value (e.g., MSE value) used, i.e., multiplied ...
AZ123's user avatar
  • 11

1
2 3 4 5
7