Questions tagged [backpropagation]
Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.
309
questions
0
votes
0
answers
37
views
Doubts on a custom loss function for regression problems
From what I read, I know we don't use log loss or cross entropy for regression problems. However, the entire logic behind binary cross entropy(say) is to firstly squeeze the y_hat between 0 and 1 (...
- 23
1
vote
0
answers
17
views
Are "textbook backpropagation" still relevant?
The above backpropagation algorithm is taken from Shalev Shwartz and Ben-David's textbook: Understanding Machine Learning. This algorithm is described in the same way as the one in Mostafa's textbook, ...
- 119
0
votes
0
answers
12
views
Is the semicolon (;) notation used to indicate operations are performed concurrently in backpropagation algorithm by Bengio?
I am trying to understand the backpropagation algorithm in a multi-layer perceptron environment.
Algorithm 6.4 Backward computation for the deep neural network of
algorithm 6.3, which uses, in ...
0
votes
0
answers
19
views
Relu derivative value
I have a stupid question on the derivative of relu activation function. After the finding the difference of the true output $t_k$ and predicted output $a_k$, why is the value of the $d_{a3}$ \ $d_{z3}$...
- 11
0
votes
1
answer
46
views
This simple python Feed forward Neural Network isn't learning. What am I doing wrong?
The backpropagation procedure is taken from the approach outlined in here.
Here is the code, commented:
...
- 103
0
votes
1
answer
44
views
How is the backward propagation is done in pytroch? When to use torch.no_grad, also when and where is the gradinte calcuated?
I have this training loop in pytorch. the loss_fn = nn.CrossEntropyLoss() and optim = torch.optim.Adam(net.parameters(), lr=lr)
<...
0
votes
0
answers
10
views
Why does weight decay produce regularisation in Deep Neural Network?
Weight decay penalizes the model to have smaller weights but how does this help in regularisation?
Any intuition on smaller weights => simpler model?
1
vote
0
answers
26
views
Confusion over taking gradients in Variational Autoencoder (VAE)
I am confused as to when to hold certain parameters constant in a VAE. I will explain with a concrete example.
We can write $\operatorname{ELBO}(\phi, \theta) = \mathbb{E}_{q_{\phi}(z)}\left[\log
\...
- 11
0
votes
1
answer
27
views
Backpropagation for each dimension of output in Pytorch
In PyTorch when we call loss.backward() it performs backpropagation for the sample (for stochastic case). Let’s consider my output is 50 dimensional. I have two loss components. First one is an array ...
- 101
2
votes
1
answer
292
views
Gradients of lower layers of NN when gradient of an upper layer is 0?
Say we have a neural network with an input layer, a hidden layer and an output layer.
Say the gradients with respect to the weights and biases of the output layer are all 0.
Then, by backpropagation ...
- 147
0
votes
0
answers
44
views
Backpropagation and Gradient Descent: Questions on math behind it
I watched this video which goes over backpropagation calculus and read the Wikipedia page on it. This is my understanding of the equations for the algorithm. I have questions regarding the equations ...
- 101
0
votes
1
answer
28
views
Problem for a math formula in Weight Uncertainty in Neural Network
I am studying the paper https://arxiv.org/pdf/1505.05424.pdf and there is a formula I don't get page 4:
I don't understand how they obtain this formula. Moreover, with chain rule, I get $\frac{\...
- 1
0
votes
1
answer
28
views
Neural Nets: Difference between activation and activation function, error on Wikipedia?
I'm reading the Wikipedia page on backpropagation and have some questions about the following equations:
$$
\frac{d C}{d a^L}\cdot \frac{d a^L}{d z^L} \cdot \frac{d z^L}{d a^{L-1}} \cdot \frac{d a^{L-...
- 101
0
votes
0
answers
21
views
How to backpropagate transposed convolution with stride and padding
Please, help! I have deadlines and I do not have time to figure out the topic on my own.
And now about the problem. I'm currently trying to figure out back propagation in transposed convolution. I ...
- 1
0
votes
0
answers
19
views
Machine Learning Geographical Data with NaNs
I have some features (physical properties related to geographical etc. of the Earth), I have a target that I'd like to predict. Sometimes this target is covered with clouds so I cannot see its actual ...
- 91
0
votes
0
answers
26
views
How to backpropagate transposed convolution?
I'm currently learning Convolutional Neural Networks and am stuck on trying to figure out how to compute gradients in a layer that uses transposed convolution. Also, how do I calculate the gradients ...
- 1
0
votes
0
answers
23
views
How does relu appears in first layer gradient of backpropagation?
I'm following Stanford's Natural language processing course in Coursera. I'm learning about "Continuous bag of words" model Where neural network with one relu(first layer) and one softmax(...
0
votes
0
answers
65
views
Is there any reference about backpropagation of the Transformer's multi-head layer?
Is there any reference about backpropagation of the Transformer's multi-head layer or multi-head attention (MHA)? I have searched various journals but have not found one yet.
- 13
1
vote
2
answers
2k
views
What exactly is Gradient norm?
I found that there is no common resource and well defined definition for "Gradient norm", most search results are based on ML experts providing answers which involves gradient norm or papers ...
- 11
1
vote
1
answer
28
views
GAN Output Gradient Calculation
Loss function for discriminator, which needs to be maximized: -log(D(x)) + log(1-D(G(z))).
Loss function for generator, which needs to be maximized: log(D(G(z)))
What would the calculation of the loss ...
- 11
0
votes
1
answer
49
views
How exactly does the mini batch method work?
I mean let's say I have a mini batch, I take an example from it and for it I do the following:
I do forward propagation.
Using the output after forward propogation - I calculate the gradients of the ...
- 11
0
votes
0
answers
28
views
How to find the derivative of the hidden state of recurrent neural networks?
Recently I am reading the following paper (link)
...
- 101
0
votes
0
answers
13
views
forward or reverse accumulation DL frameworks
Automatic differentiation can be accomplished using forward or reverse accumulation.
Quoting Wikipedia :
which mode is used in DL frameworks is used for implementation and why?
Does it have any ...
0
votes
0
answers
32
views
Backpropagation of position-wise feedforward neural network
I have read a paper entitled "Attention is all you need" by Vaswani et al. (2017). This paper use the so-called position-wise feedforward neural network, where the input of this network is a ...
- 13
0
votes
0
answers
16
views
Is It Fair to Simplify Gradient Descent as Derivative of Loss Function * Derivative of Activation Function * Derivative of Neuron?
Recently I've been trying to think of an easier way to wrap my head around gradient descent rather than just manually derive loss function with respect to weights (and chain and yada yada yada). I was ...
- 1
0
votes
3
answers
127
views
Regression model doesn't handle negative values
I'm trying to create a model that, given a feature $x_i$, predicts $y_i$ such that $y_i=ax^2_i+bx_i+c$ by using backpropagation.
To do this, I'm using the ReLU activation function for each layer.
The ...
- 152
0
votes
1
answer
16
views
moments of weight vectors in Adam
When performing backpropagation with Adam algorithm, are the moment and the second moment of the weight vectors calculated also for the weights in hidden layers?
- 152
0
votes
1
answer
36
views
Are some weight gradients equal?
I want to create a 3 layers neural network from scratch to perform linear regression.
The first and the second layer have 2 neurons, and the last layer has one neuron.
Feature vector x is divided into ...
- 152
1
vote
0
answers
23
views
How to calculate gradient of MSE in backpropagation? [duplicate]
I want to implement a neural network from scratch to solve linear regression by using backpropagation. I don't understand how to compute the gradient of the MSE cost function with respect to each ...
- 152
0
votes
0
answers
51
views
Bugs in the backpropagation algorithm in Python
I've been trying to create a simple Neural Network from scratch with a backpropagation algorithm to predict the next number based on 3 previous numbers. But for some reasons, MSE(Mean Squared Error) ...
0
votes
0
answers
24
views
Back Propagation on arbitrary depth network with ReLu
I am implementing a neural network of arbitrary depth with an arbitrary number of nodes on each depth. My forwards propagation thus looks like this (For 2 hidden layers)
...
0
votes
0
answers
62
views
How do i calculate 1 iteration of the backpropagation algorithm on this exercise
Hello, i'm currently learning Neural Networks, so have a lot of doubts on how do i calculate the weights after one iteration.
The step function for neuron 1 is:y1 = step(x1 + x2 -1.5)
The step ...
0
votes
0
answers
40
views
I am creating an multilayer LSTM model from scratch and everything seems to be mathematically correct however the model refuses to learn
I am creating the LSTM with just numpy and plotting the loss with pyplot. I have checked the derivatives again and again however have not found a mistake. The entire code with the main function can be ...
0
votes
0
answers
19
views
Function to watch out to avoid infinity during back propagation
Math functions can cause infinity during back propagations, e.g. derivative of sqrt(x) can be infinitive when x==0.
$$\frac {d}{...
- 575
0
votes
0
answers
18
views
How to derive expression for gradient in BPPT
I have the following problem: I am trying to derive final expressions for error gradients in a simple recurrent neural network (Backpropagation through Time, BPPT). The parameters and state update ...
0
votes
1
answer
95
views
GAN Generator Backpropagation Gradient Shape Doesn't Match
In the TensorFlow example (https://www.tensorflow.org/tutorials/generative/dcgan#the_discriminator) the discriminator has a single output neuron (assume batch_size=1).
Then over in the training loop ...
- 3
0
votes
1
answer
97
views
Can a Simple Neural Network Predict a 0 or 1 Output by Looking Only at the Last Input?
I wrote a simple neural network that functions similarly to many of the C# examples I've seen online. It uses weights and biases and can be trained using backpropagation. It works well for ...
- 101
0
votes
1
answer
83
views
How does Back Propagation in a Neural Net Work?
I understand that, in a Neural Net, Back Propagation is used to update the model's weights and biases to lower loss, but how does this process actually work?
- 617
0
votes
0
answers
19
views
How does backpropagation through accuracy work?
I'm using a specific constraint on my predicted logits and adding it to the loss.
In a nutshell, this constraint tries to minimize cross-overlap between the channels of my predictions. I'm using ...
- 13
0
votes
0
answers
61
views
Why backpropagation is done in every epoch when loss is always scalar?
I understand the backpropagation algorithm that it calculates the derivate of loss with respect to all the parameters in the neural network. My question is this derivate is constant right because the ...
- 101
0
votes
1
answer
53
views
How is loss calculated in truncated BPTT, for a many to one problem?
In many resources I refered to such as Justin Johnson's Lecture 12 on RNN, truncated BPTT is explained as the process of feedforward and backpropagate for smaller chunks of the sequence. These ...
- 13
3
votes
1
answer
100
views
Why is it an advantage "that Markov chains are never needed" to obtain gradients?
In the original GAN (Generative Adversarial Network) paper, Generative adversarial networks by I. Goodfellow, J. Pouget-Abadie, M. Mirza et. al. they state an advantage of the GAN is "that Markov ...
- 33
1
vote
1
answer
59
views
Is backpropagation applied every layer the same?
For example, I have layers that are pretrained. But while predicted, the loss is very high. But not because of pre-trained layers. Because of not pretrained layers. Will every layer be affected by ...
- 91
0
votes
1
answer
210
views
Can MLP model sequential data?
When modeling sequential data, RNNs are introduced as an improvement of MLP as they can model the time dependency between the inputs. It is said that feeding the last N data points in the sequence to ...
0
votes
1
answer
101
views
How can I make model unlearn? reverse backpropagation?
I stumbled upon a highly dimensional minimum that I can't seem to reproduce no matter how many hundreds of models I train.
The problem is that I went a few epochs too far and overfit on the training ...
- 519
0
votes
1
answer
55
views
Back propagation matrix shape error using Python
I wanna implement the back-propagation algorithm in python with the next code
...
- 3
0
votes
0
answers
132
views
calculating derivative of bias in backpropagation
Looking at the algorithm in wikipedia, we can implement backpropagation by calculating:
$$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$
(where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
- 101
0
votes
1
answer
14
views
evaluation of gradient for the subset of parameters using backpropagation
Consider simple feed-forward neural network with few layers.
I would like to evaluate only gradients of one particular layer, denoted by X.
This should be performed repetitively, while parameters of ...
- 1
0
votes
0
answers
13
views
Backtracking filter coefficients of Convolutional Neural Networks
I'm starting to learn how convolutional neural networks work, and I have a question regarding the filters. Apparently, these are randomly generated when the model is generated, and then as the data is ...
0
votes
0
answers
78
views
Is Loss value (e.g., MSE loss) used in the calculation for parameter update when doing gradient descent?
My question is really simple. I know the theory behind gradient descent and parameter updates, what I really haven't found clarity on is that is the loss value (e.g., MSE value) used, i.e., multiplied ...
- 11