All Questions
Tagged with neural-network backpropagation
168 questions
2
votes
1
answer
58
views
Why are the second-order derivatives of a loss function nonzero when linear combinations are involved?
I'm working on implementing Newton's method to perform second-order gradient descent in a neural network and having trouble computing the second order derivatives. I understand that in practice, ...
- 21
0
votes
0
answers
8
views
Keep Error Gradient From Being to High Torwards Input Levels
During Gradient Descent, after the error goes from each neuron down to the input layer, it gets really high. How do I fix this?
0
votes
1
answer
47
views
My custom neural network is converging but keras model not
in most cases it is probably the other way round but...
I have implemented a basic MLP neural network structure with backpropagation. My data is just a shifted quadratic function with 100 samples. I ...
1
vote
1
answer
57
views
CS 224N Back Propagation and Margin Loss in Neural Networks
I was going through Stanford CS 224 lecture notes on Back propagation.
Page 5 states:
We can see from the max-margin loss that:
∂J /∂s = − ∂J/∂s(c) = −1
I'm not sure I understand why this is the ...
- 113
0
votes
0
answers
178
views
Runtime Error: one of the variables needed for gradient computation has been modified by an inplace operation:
I have the following code for a reinforcement learning using proximal policy optimization. It gives the following run time error.
...
- 37
1
vote
0
answers
21
views
Are "textbook backpropagation" still relevant?
The above backpropagation algorithm is taken from Shalev Shwartz and Ben-David's textbook: Understanding Machine Learning. This algorithm is described in the same way as the one in Mostafa's textbook, ...
- 119
0
votes
1
answer
61
views
This simple python Feed forward Neural Network isn't learning. What am I doing wrong?
The backpropagation procedure is taken from the approach outlined in here.
Here is the code, commented:
...
- 103
0
votes
1
answer
291
views
How is the backward propagation is done in pytroch? When to use torch.no_grad, also when and where is the gradinte calcuated?
I have this training loop in pytorch. the loss_fn = nn.CrossEntropyLoss() and optim = torch.optim.Adam(net.parameters(), lr=lr)
<...
2
votes
1
answer
513
views
Gradients of lower layers of NN when gradient of an upper layer is 0?
Say we have a neural network with an input layer, a hidden layer and an output layer.
Say the gradients with respect to the weights and biases of the output layer are all 0.
Then, by backpropagation ...
- 147
0
votes
1
answer
48
views
Neural Nets: Difference between activation and activation function, error on Wikipedia?
I'm reading the Wikipedia page on backpropagation and have some questions about the following equations:
$$
\frac{d C}{d a^L}\cdot \frac{d a^L}{d z^L} \cdot \frac{d z^L}{d a^{L-1}} \cdot \frac{d a^{L-...
- 101
4
votes
2
answers
8k
views
What exactly is Gradient norm?
I found that there is no common resource and well defined definition for "Gradient norm", most search results are based on ML experts providing answers which involves gradient norm or papers ...
- 43
0
votes
1
answer
124
views
GAN Generator Backpropagation Gradient Shape Doesn't Match
In the TensorFlow example (https://www.tensorflow.org/tutorials/generative/dcgan#the_discriminator) the discriminator has a single output neuron (assume batch_size=1).
Then over in the training loop ...
- 3
0
votes
1
answer
315
views
Can a Simple Neural Network Predict a 0 or 1 Output by Looking Only at the Last Input?
I wrote a simple neural network that functions similarly to many of the C# examples I've seen online. It uses weights and biases and can be trained using backpropagation. It works well for ...
- 101
0
votes
1
answer
270
views
How does Back Propagation in a Neural Net Work?
I understand that, in a Neural Net, Back Propagation is used to update the model's weights and biases to lower loss, but how does this process actually work?
- 671
0
votes
0
answers
107
views
Why backpropagation is done in every epoch when loss is always scalar?
I understand the backpropagation algorithm that it calculates the derivate of loss with respect to all the parameters in the neural network. My question is this derivate is constant right because the ...
- 101
0
votes
1
answer
106
views
Back propagation matrix shape error using Python
I wanna implement the back-propagation algorithm in python with the next code
...
- 3
0
votes
0
answers
149
views
calculating derivative of bias in backpropagation
Looking at the algorithm in wikipedia, we can implement backpropagation by calculating:
$$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$
(where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
- 101
0
votes
1
answer
16
views
evaluation of gradient for the subset of parameters using backpropagation
Consider simple feed-forward neural network with few layers.
I would like to evaluate only gradients of one particular layer, denoted by X.
This should be performed repetitively, while parameters of ...
- 1
1
vote
0
answers
27
views
Backtracking filter coefficients of Convolutional Neural Networks
I'm starting to learn how convolutional neural networks work, and I have a question regarding the filters. Apparently, these are randomly generated when the model is generated, and then as the data is ...
- 113
0
votes
0
answers
94
views
Is Loss value (e.g., MSE loss) used in the calculation for parameter update when doing gradient descent?
My question is really simple. I know the theory behind gradient descent and parameter updates, what I really haven't found clarity on is that is the loss value (e.g., MSE value) used, i.e., multiplied ...
- 11
1
vote
0
answers
244
views
How to plot the computational graph and derive the update procedure of parameters using the backpropagation algorithm?
Please help me to solve this problem without a code (ps: this is a written problem):
Given the following loss function, please plot the computational graph, and derive the update procedure of ...
- 21
1
vote
0
answers
64
views
How to compute backpropagation gradient according chain rule for using vector/matrix differential?
I have some problems for computing derivative for sum of squares error in backprop neural network.
For example, we have a neural network as in picture. For drawing simplicity, i've dropped the sample ...
1
vote
0
answers
100
views
Help Creating a XOR Neural Network in Java?
I have been trying to create a neural network in Java, but it doesn't quite work as intended. I am using a XOR test before I move on to more advanced problems, and it doesn't seem to be learning much....
- 11
1
vote
0
answers
70
views
Is my calculation of the partial derivative of the cost function with respect to a single weight in the first layer correct?
I'm trying to understand the chain rule of backpropagation. This is what I understood. Is it correct?
$$ \frac{\partial E }{ \partial w} = \sum_{i} \frac{\partial E }{ \partial a_i^{(l)} } (\sum_{j} \...
2
votes
0
answers
103
views
Derive backpropagation for PreLU
I want to derive the back propagation functions for the Parametric Relu activation function which is defined as follows:
$$
h_a(x) = \text{max}(ax, x)
$$
I want to derive $ \frac{\partial L}{\partial ...
- 21
3
votes
1
answer
186
views
What changes is the Neural Network back-propagation algorithm doing on the weights?
I have seen the formula for back-propagation algorithm for neural network error minimization, but I am not quite sure about what changes it is performing on the weights individually.
Let us suppose a ...
user127052
1
vote
0
answers
170
views
Jacobian calculation: Partial derivatives of network outputs respect to inner layers in feedforward neural network
I'm having hard times in deriving a Jacobian (derivative of final network outputs respect to all network parameters!) for a neural network that you see in the picture below. It's about two level ...
- 63
2
votes
1
answer
1k
views
Force neural network to only product positive values
I have a custom neural network that has been written from scratch in python and also a dataset where negative target/response values are impossible, however my model sometimes produces negatives ...
- 121
1
vote
1
answer
69
views
How to train a deep neural network to return the input as it is?
The task is to train a neural network to return the input as it is, like X -> X or Y -> Y. The network should contain at ...
- 111
1
vote
1
answer
370
views
Implementing computational graph and autograd for tensor and matrix
I am trying to implement a very simple deep learning framework like PyTorch in order to get a better understanding of computational graphs and automatic differentiation. I implemented an automatic ...
3
votes
1
answer
103
views
Recurrent Neural Network (RNN) Vanishing gradient problem - Why does it affect earlier timesteps more?
I understand the concept of backpropagation in standard neural networks and backpropagation through time with RNNs, why this causes exponentially smaller gradients at earlier time steps and most of ...
- 33
0
votes
1
answer
41
views
Differentiating vector with different operation on each elements
I have some idea about how backpropagation would work for a loss function like:
loss=summation(predicted-true)^2
Where predicted and true are vectors of the same ...
- 113
1
vote
0
answers
90
views
Dummy Variables of Weights in RNN Backpropagation Through Time
In the deep learning book RNN chapter (https://www.deeplearningbook.org/contents/rnn.html), it is mentioned that -
To resolve this ambiguity, we introduce dummy variables $W^{(t)}$ that are defined to ...
- 111
0
votes
1
answer
67
views
Vanishing gradient problem
In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers?
1
vote
0
answers
61
views
Padding in Convolution Formula
Why is it that the formula for each element in a convolution between an image $I$ and a $k \times k$ sized kernel $K$ is
$$ (I*K)_{ij}=\sum_{m=0}^{k-1}\sum_{n=0}^{k-1}I_{(i-m),(j-n)}K_{mn}=\sum_{m=0}^{...
1
vote
0
answers
129
views
Tensor Backpropagation
I tried to make simple neural network layers as the following, including forward and backward propagation. Here is my reference.
Firstly I assume an one layer FC:
$Y = X \cdot W + B$
X is input, which ...
- 11
2
votes
1
answer
531
views
Gradient passthough in PyTorch
I need to quantize the inputs, but the method (bucketize) I need to do so is indifferentiable. I can of course detach the tensor, but then I lose the flow of gradients to earlier weights. I guess ...
- 203
1
vote
1
answer
1k
views
Problem with convergence of ReLu in MLP
I created neural network from scratch in python using only numpy and I'm playing with different activation functions. What I observed is quite weird and I would love to understand why this happens.
...
2
votes
0
answers
476
views
How to implement a cascaded neural network in Keras where 1st NN's forward output is cloned twice to perform forward on the 2nd NN?
Using Keras, I am trying to reproduce a few basic results from a published paper.
In this task, there are two neural networks - A & B, that are connected in a cascade formation, i.e. the output of ...
- 21
2
votes
3
answers
666
views
Neural Network Loss Function - Mean Square Error: questions about what 'n' signifies
I'm very new to neural networks and have recently learnt about the loss functions used with neural networks.
This question is in regards to the mean square error metric, defined as (from the textbook ...
- 29
3
votes
1
answer
875
views
Why the sigmoid activation function results in sub-optimal gradient descent?
I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
- 33
2
votes
1
answer
619
views
Why is it okay to set the bias vector up with zeros, and not the weight matrices?
We do not initialize weight matrices with zeros because the symmetry isn’t broken during the backward pass, and subsequently in the parameter updating process.
But it is safe to set the bias vector up ...
- 280
1
vote
0
answers
1k
views
Backpropagation Mathematics with Sigmoid Output Activation and Cross Entropy Loss
I am deriving a Weight update for a simple toy network with a Sigmoid Output Layer. I need some help double checking my math to make sure I did it correctly.
I am using Cross-Entropy Loss as my Loss ...
- 159
3
votes
0
answers
84
views
How is the input gate in the LSTM learn?
How is the input gate neural network trained what to remember by propagating the error rate from predicting the next word in the language model? How does it help it to learn if it remembered the right ...
- 131
0
votes
1
answer
45
views
Back Propagation Vs Learning rate in Neuralnet Optimisation
I was doing some research on how backpropagation works? I read that, backpropagation is used to find the optimal weight of each neuron after every iteration using partial derivates and updates the ...
- 153
7
votes
1
answer
360
views
Convolution backpropagation
I'm in the progress to learn, and understand different neural networks. I pretty much understand now feed-forward neural networks, and back-propagation of them, and now learning convolutional neural ...
- 109
0
votes
1
answer
314
views
Why my weights are being the same?
To understand how neural networks, and backpropagation are actually working, I've built a small program to do the calculations, but something is definitely wrong, as my weights are the same after ...
- 109
0
votes
3
answers
1k
views
In a neural network, is it possible to gradient descent with more than one input?
I went through a few tutorials, examples recently, and all (not sure if just for demonstration purposes) done gradient descent for one input. To get a deep understanding of backpropagation, I wrote a ...
- 109
2
votes
0
answers
54
views
Help required in understanding how the error of a convolutional layer is calculated when filter and delta of next layer have differing dimensions
I am trying to implement a CNN in NumPy so as to better understand its inner workings
My architecture is as follows
10 images with 1 channel and with 28-pixel rows and columns (Dimension : (...
- 121
1
vote
1
answer
1k
views
How does Pytorch deal with non-differentiable activation functions during backprop?
I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation ...
- 123