Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [backpropagation]

Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.

0
votes
1answer
24 views

How does back propagation works through layers like maxpooling and padding? [duplicate]

I know back propagation takes derivatives (changing one quantity wrt other). But how this is applied when there is maxpooling layer in between two Conv2D layers? How it gains its original shape when ...
0
votes
0answers
14 views

NN with gradient approximations instead of backpropagation

I've built an NN in matlab using fminunc as my optimizer for my cost function. This optimizer has the ability to approximate gradients such that the user doesn't need to supply gradients (unlike ...
0
votes
0answers
18 views

Why derivatives are accumulated over all examples in back propagation?

In back propagation, why do we accumulate dJ/dW and dJ/db over all examples (J is cost). I understand the mathematics of back ...
0
votes
0answers
10 views

Would traditional ML approaches benefit from a feedback loop (a la backprop)?

Would classical ML approaches such as SVMs, RF etc. benefit by having some sort of feedback mechanism (similar to backpropagation in Deep learning)? If so, how can one go about it? Is there any ...
1
vote
1answer
39 views

Backpropagation for noobs

I am trying to understand neural networks and how they work, by programming my own one from scratch in nodejs. Currently, i managed to build a network, that has weights, layers and neurons. I also ...
4
votes
1answer
94 views

What do “compile” , “fit” and “predict” do in Keras sequential models?

I am a little confused between these two parts of Keras sequential models functions. May someone explains what is exactly the job of each one? I mean ...
2
votes
0answers
55 views

Pytorch: How to create an update rule the doesn't come from derivatives?

I want to implement the following algorithm, taken from this book, section 13.6: Here, the neural networks' outputs are $V(S, w)$ and $\pi(A|S,\theta)$, parameterized by $w$ and $\theta$ respectively....
1
vote
1answer
15 views

quest for online tutorials on CNN backprop

I'm looking for video or text tutorial materials on CNN backprop that show how weights (convolutional kernels and last layer weights) are trained. Anyone know?
2
votes
1answer
42 views

How does Gradient Descent and Backpropagation work together?

Please forgive me as I am new to this. I have attached a diagram trying to model my understanding of neural network and Back-propagation? From videos on coursera and resources online I formed the ...
0
votes
0answers
15 views

Python LSTM Back propagation doesn't pass gradient check

I am trying to code a Recurrent neural network in python and I am having trouble getting the back propagation step to correctly calculate the gradients as when I check it using gradient checking the ...
1
vote
0answers
19 views

Backpropagation using only numpy

I've been trying to implement the backpropagation algorithm using only numpy, I've already done the Keras version, but when implementing the numpy version, the loss is diverging as seen in the image ...
0
votes
0answers
34 views

Backpropagation through LSTM and MLP layers

For didactic reason, I am currently implementing in numpy an LSTM network for classifications. I need to add on top of the LSTM another fully connected layer, because I don't want the output to have ...
1
vote
1answer
36 views

Understanding backprop for softmax

I'm looking on a given solution of the first assignment of cs231n course. Down below a snippet from the loss function. I don't really understand lines 140-143. Can ...
0
votes
1answer
27 views

Information about LTSM RNN backpropagation algorithm

I am attempting to make a LTSM RNN in python from scratch and I have completed the code for forward pass but I am struggling to find a clear outline of the equations I need to calculate to get the ...
-1
votes
0answers
17 views

Loss at each layer of neural network in linear time?

Let's say I have a neural network with $n$ inputs and $n$ layers (yes, this model is very deep), each with $n$ neurons. Each layer has the same number of neurons, and it is sparsely connected. At ...
0
votes
1answer
31 views

What is the difference between reconstruction vs backpropagation?

I was following a tutorial on understanding Restricted Boltzmann Machines (RBMs) and I noticed that they used both the terms reconstruction and ...
1
vote
0answers
40 views

Using machine learning algorithm to approximate a matrix? [closed]

I don't know if this is a good question. I would like to use machine learning to approximate a matrix. For example, a very simple one as \begin{align} A=\begin{pmatrix} 1 &0 \\ 0& 2 \end{...
0
votes
1answer
57 views

LSTM - divide gradients by number of timesteps IMMEDIATELY or in the end?

From this answer I know that the gradient of an average of many functions, is equal to the average of the gradients of those functions taken separately. The error gradient that you want to ...
0
votes
1answer
159 views

CNN backpropagation between layers

I have this CNN architecture: I know how to calculate error for weights based on the output and update weights between output<-->hidden and hidden<-->input layers. The problem is that I have ...
1
vote
1answer
34 views

Do we need to add the sigmoid derivative term in the final layer's error value?

I have been studying professor Andrew Ng's Machine Learning course on Coursera. Currently, I am trying to prove the formulas for backpropagation, which is mentioned in Week 5 (in this document). ...
0
votes
2answers
44 views

Using multiple machine learning algorithms together [closed]

I'm kinda new to machine learning and wanted to know if we could use multiple machine learning algorithms, for example, SVM and backpropagation together to solve a particular problem.
0
votes
1answer
40 views

How can I perform backpropagation directly in matrix form?

I had made a neural network library a few months ago, and I wasn't too familiar with matrices. So, instead of performing matrix dot products (between weights and inputs, then adding a bias matrix), I ...
6
votes
2answers
267 views

An error with respect to filter weights in CNN during the backpropagation

Let's say a convolutional layer takes an input $X$ with dimensions of 5x100x100 and applies 10 filters $F$ 5x5x5, thus produces an output $O$ 10 feature maps 96x96. During the backpropagation the ...
1
vote
1answer
28 views

has number of output layer of DNN any effect in speed of find the optimal answer of DNN?

has number of output layer of DNN any effect in speed of find the optimal answer of DNN? For instance the more episodes is needed to train a DNN when the number of outputs is more? Is it correct?
1
vote
0answers
280 views

Find partial derivative of softmax w.r.t logits in python

I have trouble implementing back propogation for multi class classification My neural network has 2 layers Forward propagation X -> L1 -> L2 weights W are initialized as random ...
2
votes
1answer
72 views

Backpropagation

I use chain rule when doing backpropagation and then I do Gradient Descent with weighting coefficient and I am updating the weight, so I do not understand how the method works in the equations below. ...
5
votes
1answer
77 views

Policy Gradients - gradient Log probabilities favor less likely actions?

Assume we work with neural networks, with the policy gradients method. The gradient w.r.t to the objective function $J$, is an expectation. In other words, to get this gradient $\nabla_{\theta} J(\...
2
votes
1answer
152 views

Purpose of backpropagation in neural networks

I've just finished conceptually studying linear and logistic regression functions and their optimization as preparation for neural networks. For example, say we are performing binary classification ...
5
votes
0answers
148 views

Understanding Timestamps and Batchsize of Keras LSTM considering Hiddenstates and TBPTT

What I'm trying to do What I am trying to do is predicting the next data-point $x_t$ for each point in the timeseries $[x_0, x_1, x_2,...,x_T]$ in the context of a date-stream in real-time, in theory ...
2
votes
1answer
26 views

How do you generalize from stochastic to batch learning using gradient descent?

I understand pretty OK how to derive the formulas and implement stochastic gradient descent for a deep neural network (even though the total derivative magic for hidden layers is a bit pushing my ...
1
vote
0answers
9 views

Simplifying gradients of weights (RNN)

I understand that these are the gradients of the weights/biases in an RNN (correct me if I am wrong): This is a lot to compute and I’m aware that these equations can be simplified for ease of use. ...
2
votes
1answer
57 views

Backpropogation through Maxpool and relu

Why is backpropagation through maxpool and relu needed? Purpose of backpropagation is to update weights while on the other hand maxpool and relu only perform a simple operation on the input. They don'...
1
vote
1answer
43 views

Inconsistency in training Iris dataset

I am a noob in the field of ML. I have been trying to classify the iris dataset. I managed to do it with backpropagation, and with 3 neurons in the hidden layer. But the mean square error that I get ...
1
vote
1answer
52 views

General equation - calculating backpropagation [closed]

How to calculate new weights for neurons - what is the general equation for it?
0
votes
1answer
77 views

Backpropagation - simplest explanation

Could you please explain in simplest way the algorithm (mathematical equation) of back-prop? I read lot of articles, I know for what it is, and I understand the intuition behind it, but I still don't ...
4
votes
1answer
49 views

How does backpropogation behave when there is a dropout in FC layer?

I read about dropout and how it helps in catering overfitting. In simple layman terms, it randomly drops some of the neurons in forward propagation. My question is that since these neurons will be 0 ...
0
votes
1answer
23 views

Questions about Neural Network training (back propagation) in the book PRML (Pattern Recognition and Machine Learning)

I am reading Chapter 5 of PRML. Some symbols don't seem to be clear to me. In page 243, for the chain rule for partial derivative $\dfrac{\partial E_n}{\partial w_{ji}}=\dfrac{\partial E_n}{\partial ...
1
vote
0answers
61 views

Backpropogation “The hardest derivation I have seen in machine learning”?

In week 3 of Andrew Ng's "Neural Networks and Deep Learning" on Coursera there is a video titled "Backpropogation Intuition" during which he gives the full vectorized backpropogation formula, and ends ...
1
vote
0answers
44 views

Using neural networks to solve polynomials

Is it possible to train a neural network to solve a polynomial equation? What about any non-linear single variable equation? Here are possible methods that I am thinking of trying: Training a neural ...
1
vote
0answers
133 views

Backpropagation of convolutional neural network - confusion

I've already seen many articles about this topic and Backpropagation In Convolutional Neural Networks by Jefkine seems to be the best. Although, as author said, For the purposes of simplicity we ...
0
votes
1answer
78 views

Algorithm for backpropagation through time

I am reading through this article trying to understand the bptt algorithm, in the context of an RNN. However there is one part I don’t understand: ...
1
vote
0answers
56 views

Understanding the text from the paper 'Efficient BackProp' by Yann LeCun

Sorry, I just started in Deep Learning, so I am trying my best not to assume anything unless I am absolutely sure. Going through comments here someone recommended this excellent paper on ...
3
votes
1answer
30 views

How is error back-propagated in a multi-layer RNN

Let's say I have a 2 layer LSTM cell, and I'm using this network to perform regression for input sequences of length 10 along the time axis. From what I understand, when this network is 'unfolded', ...
3
votes
1answer
46 views

Tensorflow Calculate error for a single neuron

I'm required to be able to calculate the error on a given neuron in a neural network using Tensorflow. Using this : ...
1
vote
1answer
33 views

“residual error” of LSTM during backprop vs usual “error”

What does the residual error mean when we are talking about LSTM? Taken from the middle of section 3 of this paper, where it says: "...of the residual error $\epsilon$" Where $s_0$ is the initial ...
2
votes
1answer
42 views

Conflicting directions of weights gradients in gradient descent?

In a typical ANN backpropagation setting, we have multiple weights and we try to reduce the loss function by calculating the gradient of the function with respect to the weights let's say w1, w2, w3 ...
0
votes
0answers
23 views

Custom crop layer derivatives

I implemented a heuristic for cropping score maps (Caffe). The illustration is as follows. The cropped score map is an input in a fully connected layer. Current implementation of backprop in the ...
3
votes
2answers
256 views

Python3 Tensorflow: Difference Between truncated_normal and uniform_normal when creating weights

So, I am new to using Tensorflow, and I am trying to create a neutral network, and in all the guides I saw one of the below lines of code was used. ...
-1
votes
0answers
16 views

Should the partials be averaged once or multiple times for backprop

I am trying to make a general purpose symbolic differentiation library, and I am not sure how to handle mini batch learning for SGD. When back-propagating, the sums of the partials are usually added ...