Questions tagged [backpropagation]
Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.
309
questions
4
votes
1
answer
73
views
Hochreiter LSTM (p. 4): Maximal values of logistic sigmoid derivative times weight
My questions follow the below page 4 excerpt from Hochreiter's LSTM paper:
If $f_{l_{m}}$ is the logistic sigmoid function, then the maximal
value of $f^\prime_{l_{m}}$ is 0.25. If $y^{l_{m-1}}$ ...
CommunityBot
- 1
3
votes
2
answers
58
views
Backpropagation with a different sized training set?
I'm trying to create a NN whose input is a (length m) array of 3d vectors $$\vec{x}_i = [x_{i,1},x_{i,2},x_{i,3}], \hspace{5mm}i=1:m $$
and whose output is a similarly sized array:
$$\vec{h}_{\theta,...
CommunityBot
- 1
1
vote
0
answers
26
views
Backpropgation for a single parameter on a rather simple network
Given the following network:
I'm asked to write the backpropagation process for the $b_3$ parameter, where the loss function is $L(y,z_3)=(z_3-y)^2$
I'm not supposed to calculate any of the weights ...
- 111
2
votes
1
answer
676
views
(Deep Learning) Backpropagation derivation from notes by Andrew NG
I am self-studying Andrew NG's deep learning course materials from the mcahine learning course (CS 229) of Stanford. The material is available here.
I have a question about the chain rule techniques ...
CommunityBot
- 1
3
votes
2
answers
207
views
How to interpret sudden jumps of improvement in training error vs. iteration?
In the Residual learning paper by He et al., there are a number of plots of training/test error vs. backprop iteration. I've only ever seen "smooth" curves on these plots, while in this paper's ...
CommunityBot
- 1
1
vote
2
answers
361
views
BPTT vs Vanishing Gradient Problem
I know that BPTT is the method to apply Back Propagation on RNN.
Which is works fine with RNN as it stops at certain point as changes approach to zero
but isn't it the exact Vanishing Gradient ...
CommunityBot
- 1
0
votes
1
answer
69
views
How exactly does the mini batch method work?
I mean let's say I have a mini batch, I take an example from it and for it I do the following:
I do forward propagation.
Using the output after forward propogation - I calculate the gradients of the ...
CommunityBot
- 1
1
vote
1
answer
1k
views
Generator losses in WGAN and potential convergence failure
I have been training a WGAN for a while now, with my generator training once in every five epochs.
I have tried several model architectures(no of filters) and also tried varying the relationship with ...
0
votes
1
answer
33
views
My custom neural network is converging but keras model not
in most cases it is probably the other way round but...
I have implemented a basic MLP neural network structure with backpropagation. My data is just a shifted quadratic function with 100 samples. I ...
- 1,340
4
votes
4
answers
446
views
Backpropagation: Relevance of the error signal of a neuron
During my quest to understand back propagation in a more rigorous approach I have come across with the definition of error signal of a neuron which is defined as follows for the $j^{\text{th}}$ neuron ...
CommunityBot
- 1
1
vote
0
answers
97
views
Back propagation to find the value of z
I hope all of you are doing well. I am a high school student, studying machine learning for my interest.I am studying the Backpropagation Algorithm in recent days. I got stuck in a problem:
After the ...
- 1
0
votes
1
answer
16
views
Why no scale parameter for skip connection addition?
For a simple skip connection $y = x@w + x$, the gradient dy/dx will be $w+1$.
$$\frac {\partial y}{\partial x} = w +1$$
Is +1 a bit too large and can it overpower $...
- 711
2
votes
1
answer
2k
views
Backpropagation of convolutional neural network - confusion
I've already seen many articles about this topic and Backpropagation In Convolutional Neural Networks by Jefkine seems to be the best. Although, as author said,
For the purposes of simplicity we ...
CommunityBot
- 1
1
vote
1
answer
44
views
CS 224N Back Propagation and Margin Loss in Neural Networks
I was going through Stanford CS 224 lecture notes on Back propagation.
Page 5 states:
We can see from the max-margin loss that:
∂J /∂s = − ∂J/∂s(c) = −1
I'm not sure I understand why this is the ...
- 1,988
0
votes
2
answers
136
views
Is the Cross Entropy Loss important at all, because at Backpropagation only the Softmax probability and the one hot vector are relevant?
Is the Cross Entropy Loss (CEL) important at all, because at Backpropagation (BP) only the Softmax (SM) probability and the one hot vector are relevant?
When applying BP, the derivative of CEL is the ...
CommunityBot
- 1
2
votes
1
answer
2k
views
Backpropagation with step or threshold activation function
I understand that gradient descent is local and it deals only with the inputs to the neuron, what it outputs and what it should output. In all I've seen, gradient descent needs the activation function ...
CommunityBot
- 1
0
votes
1
answer
67
views
Why not Back propagate through time in LSTM , similar to RNN
I'm trying to implement RNN and LSTM , many-to-many architecture. I reasoned myself why BPTT is necessary in RNNs and it makes sense.
But what doesn't make sense to me is, most of resources I went ...
- 141
0
votes
0
answers
39
views
Multiplying by Diagonal Matrix On Top of Standard Linear Regression
In minimizing
$$min_x||Ax-b||$$
where $A$ is overdetermined, one could use least squares method. However, if there is another diagonal matrix $d$ which has $k$ unique entries along the diagonal with $...
0
votes
1
answer
144
views
How can I make model unlearn? reverse backpropagation?
I stumbled upon a highly dimensional minimum that I can't seem to reproduce no matter how many hundreds of models I train.
The problem is that I went a few epochs too far and overfit on the training ...
CommunityBot
- 1
1
vote
1
answer
1k
views
Backpropagation with log likelihood cost function and softmax activation
In the online book on neural networks by Michael Nielsen, in chapter 3, he introduces a new cost function called as log-likelihood function defined as below
$$ C = -ln(a_y^L) $$
Suppose we have 10 ...
CommunityBot
- 1
0
votes
1
answer
35
views
How to prevent update a pretrained model if a model is optimized with backpropagation in Pytorch?
I use Pytorch exclusively to develop my model, and these are components in my model and how it works:
A generator
An encoder: a pretrained, and should not updated.
A loss function.
Input is passed to ...
- 1
22
votes
1
answer
3k
views
Understanding Timestamps and Batchsize of Keras LSTM considering Hiddenstates and TBPTT
What I'm trying to do
What I am trying to do is predicting the next data-point $x_t$ for each point in the timeseries $[x_0, x_1, x_2,...,x_T]$ in the context of a date-stream in real-time, in theory ...
CommunityBot
- 1
0
votes
1
answer
129
views
Back propagation on matrix of weights
I am trying to implement a Neural Network for binary classification using python and numpy only.
My network structure is as follows:
input features: 2 [1X2] matrix
Hidden layer1: 5 neurons [2X5] ...
CommunityBot
- 1
1
vote
1
answer
79
views
Transferring the hidden state of a RNN to another RNN
I am using Reinforcement Learning to teach an AI an Austrian Card Game with imperfect information called Schnapsen. For different states of the game, I have different neural networks (which use ...
CommunityBot
- 1
0
votes
1
answer
123
views
Can a Simple Neural Network Predict a 0 or 1 Output by Looking Only at the Last Input?
I wrote a simple neural network that functions similarly to many of the C# examples I've seen online. It uses weights and biases and can be trained using backpropagation. It works well for ...
CommunityBot
- 1
0
votes
0
answers
115
views
Runtime Error: one of the variables needed for gradient computation has been modified by an inplace operation:
I have the following code for a reinforcement learning using proximal policy optimization. It gives the following run time error.
...
- 37
5
votes
1
answer
4k
views
Gradient flow through concatenation operation
I need help in understanding the gradient flow through a concatenation operation.
I'm implementing a network (mostly a CNN) which has a concatenation operation (in pytorch). The network is defined ...
CommunityBot
- 1
2
votes
2
answers
4k
views
What exactly is Gradient norm?
I found that there is no common resource and well defined definition for "Gradient norm", most search results are based on ML experts providing answers which involves gradient norm or papers ...
- 26
0
votes
0
answers
49
views
Doubts on a custom loss function for regression problems
From what I read, I know we don't use log loss or cross entropy for regression problems. However, the entire logic behind binary cross entropy(say) is to firstly squeeze the y_hat between 0 and 1 (...
- 23
0
votes
1
answer
66
views
How does TensorFlow handle multiple samples?
Say the mini-batch has $N$ samples $(x, y)$,
how will tensorflow utilize this $N$ samples to train the network.
Will it do $N$ forward loop for each sample independently?
Will it do $N$ backward ...
CommunityBot
- 1
3
votes
3
answers
137
views
In Neural Nets, why Use Gradient Methods as Opposed to Other Metaheuristics?
In training deep and shallow neural networks, why are gradient methods (e.g. gradient descent, Nesterov, Newton-Raphson) commonly used, as opposed to other metaheuristics?
By metaheuristics I mean ...
- 133
1
vote
0
answers
19
views
Are "textbook backpropagation" still relevant?
The above backpropagation algorithm is taken from Shalev Shwartz and Ben-David's textbook: Understanding Machine Learning. This algorithm is described in the same way as the one in Mostafa's textbook, ...
- 119
0
votes
0
answers
12
views
Is the semicolon (;) notation used to indicate operations are performed concurrently in backpropagation algorithm by Bengio?
I am trying to understand the backpropagation algorithm in a multi-layer perceptron environment.
Algorithm 6.4 Backward computation for the deep neural network of
algorithm 6.3, which uses, in ...
0
votes
0
answers
30
views
Relu derivative value
I have a stupid question on the derivative of relu activation function. After the finding the difference of the true output $t_k$ and predicted output $a_k$, why is the value of the $d_{a3}$ \ $d_{z3}$...
- 11
0
votes
1
answer
58
views
This simple python Feed forward Neural Network isn't learning. What am I doing wrong?
The backpropagation procedure is taken from the approach outlined in here.
Here is the code, commented:
...
- 1,340
0
votes
3
answers
325
views
Regression model doesn't handle negative values
I'm trying to create a model that, given a feature $x_i$, predicts $y_i$ such that $y_i=ax^2_i+bx_i+c$ by using backpropagation.
To do this, I'm using the ReLU activation function for each layer.
The ...
0
votes
1
answer
122
views
How is the backward propagation is done in pytroch? When to use torch.no_grad, also when and where is the gradinte calcuated?
I have this training loop in pytorch. the loss_fn = nn.CrossEntropyLoss() and optim = torch.optim.Adam(net.parameters(), lr=lr)
<...
- 1,340
0
votes
0
answers
17
views
Why does weight decay produce regularisation in Deep Neural Network?
Weight decay penalizes the model to have smaller weights but how does this help in regularisation?
Any intuition on smaller weights => simpler model?
1
vote
0
answers
49
views
Confusion over taking gradients in Variational Autoencoder (VAE)
I am confused as to when to hold certain parameters constant in a VAE. I will explain with a concrete example.
We can write $\operatorname{ELBO}(\phi, \theta) = \mathbb{E}_{q_{\phi}(z)}\left[\log
\...
- 11
0
votes
1
answer
49
views
Backpropagation for each dimension of output in Pytorch
In PyTorch when we call loss.backward() it performs backpropagation for the sample (for stochastic case). Let’s consider my output is 50 dimensional. I have two loss components. First one is an array ...
- 190
2
votes
1
answer
351
views
Gradients of lower layers of NN when gradient of an upper layer is 0?
Say we have a neural network with an input layer, a hidden layer and an output layer.
Say the gradients with respect to the weights and biases of the output layer are all 0.
Then, by backpropagation ...
- 1,362
0
votes
1
answer
47
views
Problem for a math formula in Weight Uncertainty in Neural Network
I am studying the paper https://arxiv.org/pdf/1505.05424.pdf and there is a formula I don't get page 4:
I don't understand how they obtain this formula. Moreover, with chain rule, I get $\frac{\...
- 586
0
votes
0
answers
47
views
Backpropagation and Gradient Descent: Questions on math behind it
I watched this video which goes over backpropagation calculus and read the Wikipedia page on it. This is my understanding of the equations for the algorithm. I have questions regarding the equations ...
- 101
3
votes
1
answer
738
views
Why the sigmoid activation function results in sub-optimal gradient descent?
I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
CommunityBot
- 1
0
votes
1
answer
28
views
Neural Nets: Difference between activation and activation function, error on Wikipedia?
I'm reading the Wikipedia page on backpropagation and have some questions about the following equations:
$$
\frac{d C}{d a^L}\cdot \frac{d a^L}{d z^L} \cdot \frac{d z^L}{d a^{L-1}} \cdot \frac{d a^{L-...
0
votes
0
answers
37
views
How to backpropagate transposed convolution with stride and padding
Please, help! I have deadlines and I do not have time to figure out the topic on my own.
And now about the problem. I'm currently trying to figure out back propagation in transposed convolution. I ...
- 1
0
votes
0
answers
21
views
Machine Learning Geographical Data with NaNs
I have some features (physical properties related to geographical etc. of the Earth), I have a target that I'd like to predict. Sometimes this target is covered with clouds so I cannot see its actual ...
- 101
10
votes
1
answer
6k
views
Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training?
ReLU is an activation function defined as $h = \max(0, a)$ where $a = Wx + b$.
Normally, we train neural networks with first-order methods such as SGD, Adam, RMSprop, Adadelta, or Adagrad. ...
- 107
0
votes
0
answers
26
views
How to backpropagate transposed convolution?
I'm currently learning Convolutional Neural Networks and am stuck on trying to figure out how to compute gradients in a layer that uses transposed convolution. Also, how do I calculate the gradients ...
- 1
0
votes
0
answers
23
views
How does relu appears in first layer gradient of backpropagation?
I'm following Stanford's Natural language processing course in Coursera. I'm learning about "Continuous bag of words" model Where neural network with one relu(first layer) and one softmax(...
