# Questions tagged [backpropagation]

Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.

274 questions
Filter by
Sorted by
Tagged with
26 views

### Why can't a multi-layer linear neural network fit this linear function data?

I am learning to implement a neural network with gradient descent, and encountered this problem, please. Using the target function ...
14 views

15 views

### Differentiation of Learning Capabilities of Different Networks

I have a conceptual problem regarding the overall learning capability of a neural network differentiated by the different types of input that we can give to the network. Suppose that we have a ...
12 views

### Update function for NN with logistic and sofmax

Can anyone help me confirm my work or find resources on how to come up with the update function for each layer in the Neural Network for multi-class classification problem i.e I am using logistic as ...
7 views

### Novice machine learner wondering how to interpret big variance in batch error across batches in MNIST perceptron

I'm trying to get a better understanding of basic neural networks by implementing a little framework in C++. I've started with the classical MNIST exercise. I get to 91% accuracy on the test sample ...
22 views

I am trying to derive the backpropagation for a single convolutional layer (padding layer is being implemented separately, so no padding argument for the convolutional layer). This layer is given $\... 0answers 30 views ### Batch normalization backpropagation doubts I have recently studied the batch normalization layer and its backpropagation process, using as my main sources the original paper and this website showing part of the derivation process, but there is ... 3answers 263 views ### What backpropagation actually is? I have a conceptual question due to terminology that bothers me. Is backpropagation algorithm a neural network training algorithm or is it just a recursive algorithm in order to calculate a Jacobian ... 0answers 12 views ### Jacobian calculation: Partial derivatives of network outputs respect to inner layers in feedforward neural network I'm having hard times in deriving a Jacobian (derivative of final network outputs respect to all network parameters!) for a neural network that you see in the picture below. It's about two level ... 0answers 8 views ### Backpropagation in RNN in discrete visible units Refer to https://www.reddit.com/r/MachineLearning/comments/40ldq6/generative_adversarial_networks_for_text/ Goodfellow said that we still don't have a way to use GANs in NLP because of its discrete (... 0answers 10 views ### Regarding the backpropagation I was reading a GNN intro book which discussed backpropation in NNs shortly. I was particularly concerned about the following equation: This is different from what I know, that when the parameter ... 0answers 20 views ### Andrew Ng Deep Learning Gradient Descent of Softmax is just y_hat - y? At about 8:30 in the video here: https://www.youtube.com/watch?v=ueO_Ph0Pyqk so for the given example with 4 classes and first ground truth y being [0,1,0,0] and y_hat being [0.3,0.2,0.1,0.4] for ... 0answers 176 views ### PyTorch backwards() call on loss function Can someone confirm that a call to loss.backward() given loss defined with nn.MSELoss() if called in a loop like this: ... 1answer 46 views ### Force neural network to only product positive values I have a custom neural network that has been written from scratch in python and also a dataset where negative target/response values are impossible, however my model sometimes produces negatives ... 0answers 25 views ### how to calculate parameters of an RNN using backpropagation I'm trying to find out the two binary inputs are identical or not using RNN. my architecture is like this: I have the following functions: Where vT is the transpose of vector v and the activation ... 1answer 39 views ### How to train a deep neural network to return the input as it is? The task is to train a neural network to return the input as it is, like X -> X or Y -> Y. The network should contain at ... 1answer 58 views ### Implementing computational graph and autograd for tensor and matrix I am trying to implement a very simple deep learning framework like PyTorch in order to get a better understanding of computational graphs and automatic differentiation. I implemented an automatic ... 2answers 39 views ### Backpropagation During Neural Networks Training - Units while updating weights I found this article that describes how neural networks work. This paragraph near the end caught my eye and explains how weights are updated: So we see that$\theta_i := \theta_i + \nabla\theta_i$... 0answers 57 views ### How does Keras optimization for a network with multiple outputs I currently have a neural network that takes in 3 numbers as inputs and outputs 3 numbers. I've attached a picture of the network below and my code is accessible through the following link: [Google ... 1answer 730 views ### How batch normalization layer resolve the vanishing gradient problem? According to this article: https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484 The vanishing gradient problem occurs when using the sigmoid ... 0answers 9 views ### Backpropagation for Network Architecture Search (NAS) in normal backpropagation, there is only a single edge in between two nodes. However in GDAS, there are multiple parallel edges in between two nodes. So, how to perform backpropagation for Network ... 0answers 40 views ### Back propagation process of RNN? I'm learning how to use the Recurrent Neural Network model (RNN). I'm not entirely sure about the feed-forward procedure in RNN. It includes, for example, input, hidden state, and output. As far as I ... 0answers 20 views ### Back propagation to find the value of z I hope all of you are doing well. I am a high school student, studying machine learning for my interest.I am studying the Backpropagation Algorithm in recent days. I got stuck in a problem: After the ... 0answers 21 views ### Getting gradient for gradCam in pytorch I am using forward and backward hook in my pytorch densenet121 model. I set requires_grad to False at the time of training. ... 0answers 9 views ### Matrix dimensions issue on calculating Softmax Derivatives I am trying to get the matrix dimensions right for computing derivative of a two layers network where the last layer is softmax function. For simplicity I am only interested to get derivatives of W2 w.... 0answers 99 views ### Backprop with relu and softmax in matlab I am trying to implement a network for a classification task and I am kinda struggling with backpropagation. The network should classify MNIST dataset (0 - 9). As a training set, I have 4500 images of ... 1answer 38 views ### Recurrent Neural Network (RNN) Vanishing gradient problem - Why does it affect earlier timesteps more? I understand the concept of backpropagation in standard neural networks and backpropagation through time with RNNs, why this causes exponentially smaller gradients at earlier time steps and most of ... 1answer 35 views ### Differentiating vector with different operation on each elements I have some idea about how backpropagation would work for a loss function like: loss=summation(predicted-true)^2 Where predicted and true are vectors of the same ... 1answer 22 views ### Is the usage of the "momentum" significiantly superior to the conventional weight update The "momentum" adds a little of the history of the last weight updates to the actual update, with diminishing weight history (older momentum shares get smaller). Is it significiantly ... 0answers 34 views ### GRU - why gamma is close to 0 or 1 resolves the vanishing gradient problem? Background It is explained that in GRU (Gated Recurrent Unit), the gate value$\Gamma$close to zero mitigates the vanishing gradient problem. Coursera Sequence Model by Andrew Ng - Week 1 Gated ... 0answers 36 views ### Dummy Variables of Weights in RNN Backpropagation Through Time In the deep learning book RNN chapter (https://www.deeplearningbook.org/contents/rnn.html), it is mentioned that - To resolve this ambiguity, we introduce dummy variables$W^{(t)}$that are deﬁned to ... 0answers 26 views ### Gradient calculation analysis I'm using VGG16 pretrained architecture for classification and visualization of result using Guided Backpropagation technique. I have used tensorflow code for calaulating the input gradientSharing ... 1answer 28 views ### What issue is there, when training this network with gradient descent? [closed] Suppose we have the following fully connected network made of perceptrons with a sign function as the activation unit, what issue arises, when trying to train this network with gradient descent? 1answer 27 views ### Vanishing gradient problem In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers? 1answer 32 views ### Why is backpropagation used for finding the loss gradient? I am relatively new to the world of machine learning. After getting a general idea of the concept, I tried creating a program for training a deep learning network from scratch. My goal was to use as ... 0answers 172 views ### Need help to understand backpropagation for gated recurrent units (GRU) I'm stuck regarding the implementation of backpropagation in a binary classification task using a GRU. I wanted to know if someone could tell me how to proceed. I was able to understand how BP works ... 0answers 29 views ### Padding in Convolution Formula Why is it that the formula for each element in a convolution between an image$I$and a$k \times k$sized kernel$K$is $$(I*K)_{ij}=\sum_{m=0}^{k-1}\sum_{n=0}^{k-1}I_{(i-m),(j-n)}K_{mn}=\sum_{m=0}^{... 0answers 17 views ### Understanding the convolution formula According to several sources this formula, or the center originated version of it, is used to calculate an element of a convolution between an image I and a kernel K of size k \times k:$$ (I*K)... 0answers 20 views ### Decoupling the performance impact of batch sizes from their impact on training speed We are training a sequence-to-sequence-model to do something like Named Entity Recognition using PyTorch. It turns out, we get the best results with batch size 1. However, this slows down training ... 0answers 70 views ### Tensor Backpropagation I tried to make simple neural network layers as the following, including forward and backward propagation. Here is my reference. Firstly I assume an one layer FC:$Y = X \cdot W + B\$ X is input, which ...
43 views

I need to quantize the inputs, but the method (bucketize) I need to do so is indifferentiable. I can of course detach the tensor, but then I lose the flow of gradients to earlier weights. I guess ...
309 views

### Why divide by batch size when back-propagate from softmax + log loss

Question In neural network mini batch training, at the back-propagation from the (Softmax + cross entropy log loss) layer, the gradient is divided by the batch size. Please explain why need to do so. ...