Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [backpropagation]

Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

calculating gradient descent

when using mini batch gradient descent , we perform backpropagation after each batch , ie we calculate the gradient after each batch , we also capture y-hat after each sample in the batch and finally ...
user avatar
  • 13
0 votes
0 answers
7 views

Backtracking filter coefficients of Convolutional Neural Networks

I'm starting to learn how convolutional neural networks work, and I have a question regarding the filters. Apparently, these are randomly generated when the model is generated, and then as the data is ...
user avatar
0 votes
0 answers
16 views

Backpropagation in NN

During backward pass, which gradients are kept and which gradients are discarded? Why are some gradients discarded? I know that forward pass is computing the output of the network given the inputs and ...
user avatar
0 votes
0 answers
17 views

Is Loss value (e.g., MSE loss) used in the calculation for parameter update when doing gradient descent?

My question is really simple. I know the theory behind gradient descent and parameter updates, what I really haven't found clarity on is that is the loss value (e.g., MSE value) used, i.e., multiplied ...
user avatar
  • 1
1 vote
0 answers
112 views

How to plot the computational graph and derive the update procedure of parameters using the backpropagation algorithm?

Please help me to solve this problem without a code (ps: this is a written problem): Given the following loss function, please plot the computational graph, and derive the update procedure of ...
user avatar
  • 21
0 votes
0 answers
21 views

Keras Backpropagation when Later Layers are Frozen

I am working on a project with facial image translation and GANs and still have some conceptual misunderstandings. In my definition of my model, I extract a deep embedding of my generated image and ...
user avatar
0 votes
0 answers
12 views

Question about grad() from Deep Learning by Chollet

On page 58 of the second edition of Deep Learning with Python, Chollet is illustrating an example of a forward and backward pass of a computation graph. The computation graph is given by: $$ x\to w\...
user avatar
  • 101
0 votes
0 answers
11 views

Is there a difference between AutoGrad and explicit derivatives (gradient)?

Will there be some differences between applying AutoGrad on the loss function (using a python library) and applying explicit gradient (the gradient from the paper or the update rule)? For example: ...
user avatar
  • 1
0 votes
0 answers
20 views

Backpropagation in conventional recurrent neural network

I'm in the beginning to learn and understand recurrent neural networks. I am trying to understand the back-propagation process which helps us to find the gradients that are required to update the ...
user avatar
0 votes
0 answers
10 views

Is every timestep when training an LSTM part of the error, or just the end of the sequence?

I'm attempting to write my own LSTM network in C++ for fun. I've already got a regressive and classification network working with regular perceptrons and it works well. What I do currently is divide ...
user avatar
1 vote
0 answers
26 views

How to compute backpropagation gradient according chain rule for using vector/matrix differential?

I have some problems for computing derivative for sum of squares error in backprop neural network. For example, we have a neural network as in picture. For drawing simplicity, i've dropped the sample ...
user avatar
0 votes
0 answers
20 views

How to manually calculate the gradient that will propagate back over the network using the REINFORCE algorithm?

I am trying to implement deep reinforcement policy gradient REINFORCE in C++ and for my case there is no "autograd" method like in pytorch so I have to manually calculate the gradient. Let´s ...
user avatar
1 vote
0 answers
33 views

Help Creating a XOR Neural Network in Java?

I have been trying to create a neural network in Java, but it doesn't quite work as intended. I am using a XOR test before I move on to more advanced problems, and it doesn't seem to be learning much....
user avatar
  • 11
0 votes
1 answer
18 views

Can we not backpropagate model

I saw a model based on CNN for question classification. The author said that they don't backpropagate gradient to embeddings. How this is possible to update network if you don't backpropagate please? ...
user avatar
  • 179
0 votes
1 answer
107 views

Backpropagation of Bias in Neural networks

My goal is to calculate backpropagation(Especially the backpropagation of the bias). For example, X, W and B are python numpy array, such as [[0,0],[0,1]] , [[5,5,5],[10,10,10]] and [1,2,3] for each. ...
user avatar
  • 105
0 votes
0 answers
35 views

Computing the Gradients of Dense Layer

Given the following Dense layer below, I am trying to understand the backward pass. As I understand, we should have well defined loss function (sigmoid, relu, etc.) and then a pre-activation function. ...
user avatar
  • 179
1 vote
0 answers
25 views

Is my calculation of the partial derivative of the cost function with respect to a single weight in the first layer correct?

I'm trying to understand the chain rule of backpropagation. This is what I understood. Is it correct? $$ \frac{\partial E }{ \partial w} = \sum_{i} \frac{\partial E }{ \partial a_i^{(l)} } (\sum_{j} \...
user avatar
0 votes
0 answers
15 views

Back propagation with help of Taylor series in calculating derivative of loss function

I am wondering how to train a multi-layer neural network using back propagation algorithm with help of Taylor series when calculating partial derivative of loss function with respect to weights. I ...
user avatar
2 votes
0 answers
15 views

Derive backpropagation for PreLU

I want to derive the back propagation functions for the Parametric Relu activation function which is defined as follows: $$ h_a(x) = \text{max}(ax, x) $$ I want to derive $ \frac{\partial L}{\partial ...
user avatar
  • 21
3 votes
0 answers
17 views

Homework/class help: Backward propagation of max pooling if each element in an array determines more than one value?

(This isn't actually my homework, and in fact wasn't addressed in my homework, but I was confused about this because my homework hadn't addressed this) For example if I have an array: And I do max ...
user avatar
0 votes
0 answers
8 views

Back propagation - updating basis values

When doing backpropergation using gradient decent (batch descent), the dW / DL ends up with a matrix the same size a W regardless of the size of the batch. The dB / dL (basis adjustment values) ends ...
user avatar
  • 101
3 votes
1 answer
147 views

What changes is the Neural Network back-propagation algorithm doing on the weights?

I have seen the formula for back-propagation algorithm for neural network error minimization, but I am not quite sure about what changes it is performing on the weights individually. Let us suppose a ...
user avatar
0 votes
1 answer
25 views

How does TensorFlow handle multiple samples?

Say the mini-batch has $N$ samples $(x, y)$, how will tensorflow utilize this $N$ samples to train the network. Will it do $N$ forward loop for each sample independently? Will it do $N$ backward ...
user avatar
  • 101
1 vote
1 answer
51 views

Transferring the hidden state of a RNN to another RNN

I am using Reinforcement Learning to teach an AI an Austrian Card Game with imperfect information called Schnapsen. For different states of the game, I have different neural networks (which use ...
user avatar
0 votes
1 answer
25 views

Understanding intution behind sigmoid curve in the context of back propagation

I was trying to understand significance of S-shape of sigmoid / logistic function. The slope/derivative of sigmoid approaches zero for very large and very small input values. That is $σ'(z) ≈ 0$ for $...
user avatar
  • 165
0 votes
0 answers
16 views

Differentiation of Learning Capabilities of Different Networks

I have a conceptual problem regarding the overall learning capability of a neural network differentiated by the different types of input that we can give to the network. Suppose that we have a ...
user avatar
0 votes
0 answers
16 views

Update function for NN with logistic and sofmax

Can anyone help me confirm my work or find resources on how to come up with the update function for each layer in the Neural Network for multi-class classification problem i.e I am using logistic as ...
user avatar
  • 1
0 votes
0 answers
7 views

Novice machine learner wondering how to interpret big variance in batch error across batches in MNIST perceptron

I'm trying to get a better understanding of basic neural networks by implementing a little framework in C++. I've started with the classical MNIST exercise. I get to 91% accuracy on the test sample ...
user avatar
  • 121
1 vote
0 answers
30 views

Backpropagation derivation for a convolutional layer

I am trying to derive the backpropagation for a single convolutional layer (padding layer is being implemented separately, so no padding argument for the convolutional layer). This layer is given $\...
user avatar
0 votes
0 answers
44 views

Batch normalization backpropagation doubts

I have recently studied the batch normalization layer and its backpropagation process, using as my main sources the original paper and this website showing part of the derivation process, but there is ...
user avatar
3 votes
3 answers
287 views

What backpropagation actually is?

I have a conceptual question due to terminology that bothers me. Is backpropagation algorithm a neural network training algorithm or is it just a recursive algorithm in order to calculate a Jacobian ...
user avatar
  • 53
1 vote
0 answers
39 views

Jacobian calculation: Partial derivatives of network outputs respect to inner layers in feedforward neural network

I'm having hard times in deriving a Jacobian (derivative of final network outputs respect to all network parameters!) for a neural network that you see in the picture below. It's about two level ...
user avatar
  • 53
0 votes
0 answers
11 views

Backpropagation in RNN in discrete visible units

Refer to https://www.reddit.com/r/MachineLearning/comments/40ldq6/generative_adversarial_networks_for_text/ Goodfellow said that we still don't have a way to use GANs in NLP because of its discrete (...
user avatar
0 votes
0 answers
11 views

Regarding the backpropagation

I was reading a GNN intro book which discussed backpropation in NNs shortly. I was particularly concerned about the following equation: This is different from what I know, that when the parameter ...
user avatar
0 votes
0 answers
21 views

Andrew Ng Deep Learning Gradient Descent of Softmax is just y_hat - y?

At about 8:30 in the video here: https://www.youtube.com/watch?v=ueO_Ph0Pyqk so for the given example with 4 classes and first ground truth y being [0,1,0,0] and y_hat being [0.3,0.2,0.1,0.4] for ...
user avatar
1 vote
0 answers
442 views

PyTorch backwards() call on loss function

Can someone confirm that a call to loss.backward() given loss defined with nn.MSELoss() if called in a loop like this: ...
user avatar
1 vote
1 answer
93 views

Force neural network to only product positive values

I have a custom neural network that has been written from scratch in python and also a dataset where negative target/response values are impossible, however my model sometimes produces negatives ...
user avatar
  • 11
0 votes
0 answers
42 views

how to calculate parameters of an RNN using backpropagation

I'm trying to find out the two binary inputs are identical or not using RNN. my architecture is like this: I have the following functions: Where vT is the transpose of vector v and the activation ...
user avatar
1 vote
1 answer
49 views

How to train a deep neural network to return the input as it is?

The task is to train a neural network to return the input as it is, like X -> X or Y -> Y. The network should contain at ...
user avatar
  • 111
1 vote
1 answer
111 views

Implementing computational graph and autograd for tensor and matrix

I am trying to implement a very simple deep learning framework like PyTorch in order to get a better understanding of computational graphs and automatic differentiation. I implemented an automatic ...
user avatar
2 votes
2 answers
41 views

Backpropagation During Neural Networks Training - Units while updating weights

I found this article that describes how neural networks work. This paragraph near the end caught my eye and explains how weights are updated: So we see that $\theta_i := \theta_i + \nabla\theta_i$ ...
user avatar
0 votes
0 answers
80 views

How does Keras optimization for a network with multiple outputs

I currently have a neural network that takes in 3 numbers as inputs and outputs 3 numbers. I've attached a picture of the network below and my code is accessible through the following link: [Google ...
user avatar
2 votes
1 answer
2k views

How batch normalization layer resolve the vanishing gradient problem?

According to this article: https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484 The vanishing gradient problem occurs when using the sigmoid ...
user avatar
0 votes
0 answers
12 views

Backpropagation for Network Architecture Search (NAS)

in normal backpropagation, there is only a single edge in between two nodes. However in GDAS, there are multiple parallel edges in between two nodes. So, how to perform backpropagation for Network ...
user avatar
  • 101
0 votes
0 answers
41 views

Back propagation process of RNN?

I'm learning how to use the Recurrent Neural Network model (RNN). I'm not entirely sure about the feed-forward procedure in RNN. It includes, for example, input, hidden state, and output. As far as I ...
user avatar
1 vote
0 answers
25 views

Back propagation to find the value of z

I hope all of you are doing well. I am a high school student, studying machine learning for my interest.I am studying the Backpropagation Algorithm in recent days. I got stuck in a problem: After the ...
user avatar
  • 111
3 votes
1 answer
42 views

Recurrent Neural Network (RNN) Vanishing gradient problem - Why does it affect earlier timesteps more?

I understand the concept of backpropagation in standard neural networks and backpropagation through time with RNNs, why this causes exponentially smaller gradients at earlier time steps and most of ...
user avatar
0 votes
1 answer
37 views

Differentiating vector with different operation on each elements

I have some idea about how backpropagation would work for a loss function like: loss=summation(predicted-true)^2 Where predicted and true are vectors of the same ...
user avatar
  • 113
0 votes
1 answer
22 views

Is the usage of the "momentum" significiantly superior to the conventional weight update

The "momentum" adds a little of the history of the last weight updates to the actual update, with diminishing weight history (older momentum shares get smaller). Is it significiantly ...
user avatar
1 vote
0 answers
45 views

Dummy Variables of Weights in RNN Backpropagation Through Time

In the deep learning book RNN chapter (https://www.deeplearningbook.org/contents/rnn.html), it is mentioned that - To resolve this ambiguity, we introduce dummy variables $W^{(t)}$ that are defined to ...
user avatar
  • 111

1
2 3 4 5 6