# Questions tagged [backpropagation]

Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.

309 questions
Filter by
Sorted by
Tagged with
38 views

### Doubts on a custom loss function for regression problems

From what I read, I know we don't use log loss or cross entropy for regression problems. However, the entire logic behind binary cross entropy(say) is to firstly squeeze the y_hat between 0 and 1 (...
1 vote
17 views

### Are "textbook backpropagation" still relevant?

The above backpropagation algorithm is taken from Shalev Shwartz and Ben-David's textbook: Understanding Machine Learning. This algorithm is described in the same way as the one in Mostafa's textbook, ...
12 views

### Is the semicolon (;) notation used to indicate operations are performed concurrently in backpropagation algorithm by Bengio?

I am trying to understand the backpropagation algorithm in a multi-layer perceptron environment. Algorithm 6.4 Backward computation for the deep neural network of algorithm 6.3, which uses, in ...
19 views

### Relu derivative value

I have a stupid question on the derivative of relu activation function. After the finding the difference of the true output $t_k$ and predicted output $a_k$, why is the value of the $d_{a3}$ \ $d_{z3}$...
48 views

### This simple python Feed forward Neural Network isn't learning. What am I doing wrong?

The backpropagation procedure is taken from the approach outlined in here. Here is the code, commented: ...
44 views

### How is the backward propagation is done in pytroch? When to use torch.no_grad, also when and where is the gradinte calcuated?

I have this training loop in pytorch. the loss_fn = nn.CrossEntropyLoss() and optim = torch.optim.Adam(net.parameters(), lr=lr) <...
11 views

### Why does weight decay produce regularisation in Deep Neural Network?

Weight decay penalizes the model to have smaller weights but how does this help in regularisation? Any intuition on smaller weights => simpler model?
1 vote
28 views

28 views

18 views

### How to derive expression for gradient in BPPT

I have the following problem: I am trying to derive final expressions for error gradients in a simple recurrent neural network (Backpropagation through Time, BPPT). The parameters and state update ...
95 views

### GAN Generator Backpropagation Gradient Shape Doesn't Match

In the TensorFlow example (https://www.tensorflow.org/tutorials/generative/dcgan#the_discriminator) the discriminator has a single output neuron (assume batch_size=1). Then over in the training loop ...
98 views

### Can a Simple Neural Network Predict a 0 or 1 Output by Looking Only at the Last Input?

I wrote a simple neural network that functions similarly to many of the C# examples I've seen online. It uses weights and biases and can be trained using backpropagation. It works well for ...
83 views

### How does Back Propagation in a Neural Net Work?

I understand that, in a Neural Net, Back Propagation is used to update the model's weights and biases to lower loss, but how does this process actually work?
19 views

### How does backpropagation through accuracy work?

I'm using a specific constraint on my predicted logits and adding it to the loss. In a nutshell, this constraint tries to minimize cross-overlap between the channels of my predictions. I'm using ...
65 views

### Why backpropagation is done in every epoch when loss is always scalar?

I understand the backpropagation algorithm that it calculates the derivate of loss with respect to all the parameters in the neural network. My question is this derivate is constant right because the ...
53 views

### How is loss calculated in truncated BPTT, for a many to one problem?

In many resources I refered to such as Justin Johnson's Lecture 12 on RNN, truncated BPTT is explained as the process of feedforward and backpropagate for smaller chunks of the sequence. These ...
101 views

### Why is it an advantage "that Markov chains are never needed" to obtain gradients?

In the original GAN (Generative Adversarial Network) paper, Generative adversarial networks by I. Goodfellow, J. Pouget-Abadie, M. Mirza et. al. they state an advantage of the GAN is "that Markov ...
1 vote
59 views

### Is backpropagation applied every layer the same?

For example, I have layers that are pretrained. But while predicted, the loss is very high. But not because of pre-trained layers. Because of not pretrained layers. Will every layer be affected by ...
215 views

### Can MLP model sequential data?

When modeling sequential data, RNNs are introduced as an improvement of MLP as they can model the time dependency between the inputs. It is said that feeding the last N data points in the sequence to ...
103 views

### How can I make model unlearn? reverse backpropagation?

I stumbled upon a highly dimensional minimum that I can't seem to reproduce no matter how many hundreds of models I train. The problem is that I went a few epochs too far and overfit on the training ...
55 views

### Back propagation matrix shape error using Python

I wanna implement the back-propagation algorithm in python with the next code ...
132 views

### calculating derivative of bias in backpropagation

Looking at the algorithm in wikipedia, we can implement backpropagation by calculating: $$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$ (where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
14 views

### evaluation of gradient for the subset of parameters using backpropagation

Consider simple feed-forward neural network with few layers. I would like to evaluate only gradients of one particular layer, denoted by X. This should be performed repetitively, while parameters of ...