Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.

0
votes
0answers
5 views

Error with respect to kernel weights in convolutional layer

Let's say, I have a convolution layer with: 3 input channels $X$ with dimensions 3x32x32 5 kernel filters $F$ with dimensions 3x7x7 Assuming the stride = 1, the layer produces an output $O$ with ...
2
votes
1answer
14 views

Backpropogation through Maxpool and relu

Why is backpropagation through maxpool and relu needed? Purpose of backpropagation is to update weights while on the other hand maxpool and relu only perform a simple operation on the input. They don'...
1
vote
1answer
24 views

Inconsistency in training Iris dataset

I am a noob in the field of ML. I have been trying to classify the iris dataset. I managed to do it with backpropagation, and with 3 neurons in the hidden layer. But the mean square error that I get ...
1
vote
1answer
25 views

General equation - calculating backpropagation [closed]

How to calculate new weights for neurons - what is the general equation for it?
0
votes
1answer
17 views

Backpropagation - simplest explanation

Could you please explain in simplest way the algorithm (mathematical equation) of back-prop? I read lot of articles, I know for what it is, and I understand the intuition behind it, but I still don't ...
4
votes
1answer
36 views

How does backpropogation behave when there is a dropout in FC layer?

I read about dropout and how it helps in catering overfitting. In simple layman terms, it randomly drops some of the neurons in forward propagation. My question is that since these neurons will be 0 ...
0
votes
1answer
19 views

Questions about Neural Network training (back propagation) in the book PRML (Pattern Recognition and Machine Learning)

I am reading Chapter 5 of PRML. Some symbols don't seem to be clear to me. In page 243, for the chain rule for partial derivative $\dfrac{\partial E_n}{\partial w_{ji}}=\dfrac{\partial E_n}{\partial ...
1
vote
0answers
30 views

Backpropogation “The hardest derivation I have seen in machine learning”?

In week 3 of Andrew Ng's "Neural Networks and Deep Learning" on Coursera there is a video titled "Backpropogation Intuition" during which he gives the full vectorized backpropogation formula, and ends ...
0
votes
0answers
12 views

Combining ARIMA and BackPropogation for CPU Utilization

I have to implement a model combining both ARIMA and BP to predict CPU Utilization based on previous data and reduce error. After implementing this ARIMA model I'm getting MSE of 1.6. I want to reduce ...
1
vote
0answers
25 views

Using neural networks to solve polynomials

Is it possible to train a neural network to solve a polynomial equation? What about any non-linear single variable equation? Here are possible methods that I am thinking of trying: Training a neural ...
1
vote
0answers
38 views

Backpropagation of convolutional neural network - confusion

I've already seen many articles about this topic and Backpropagation In Convolutional Neural Networks by Jefkine seems to be the best. Although, as author said, For the purposes of simplicity we ...
0
votes
0answers
11 views

Keras update Layers which are only Part of the Loss Function

I'm implementing a variational inference model. There are two main structures a inference network and a generative model. The first part is only used for training and only appears in the loss function....
0
votes
1answer
20 views

Algorithm for backpropagation through time

I am reading through this article trying to understand the bptt algorithm, in the context of an RNN. However there is one part I don’t understand: ...
1
vote
0answers
20 views

Understanding the text from the paper 'Efficient BackProp' by Yann LeCun

Sorry, I just started in Deep Learning, so I am trying my best not to assume anything unless I am absolutely sure. Going through comments here someone recommended this excellent paper on ...
2
votes
1answer
17 views

How is error back-propagated in a multi-layer RNN

Let's say I have a 2 layer LSTM cell, and I'm using this network to perform regression for input sequences of length 10 along the time axis. From what I understand, when this network is 'unfolded', ...
3
votes
1answer
30 views

Tensorflow Calculate error for a single neuron

I'm required to be able to calculate the error on a given neuron in a neural network using Tensorflow. Using this : ...
1
vote
1answer
19 views

“residual error” of LSTM during backprop vs usual “error”

What does the residual error mean when we are talking about LSTM? Taken from the middle of section 3 of this paper, where it says: "...of the residual error $\epsilon$" Where $s_0$ is the initial ...
2
votes
1answer
21 views

Conflicting directions of weights gradients in gradient descent?

In a typical ANN backpropagation setting, we have multiple weights and we try to reduce the loss function by calculating the gradient of the function with respect to the weights let's say w1, w2, w3 ...
0
votes
0answers
11 views

Custom crop layer derivatives

I implemented a heuristic for cropping score maps (Caffe). The illustration is as follows. The cropped score map is an input in a fully connected layer. Current implementation of backprop in the ...
3
votes
2answers
20 views

Python3 Tensorflow: Difference Between truncated_normal and uniform_normal when creating weights

So, I am new to using Tensorflow, and I am trying to create a neutral network, and in all the guides I saw one of the below lines of code was used. ...
0
votes
0answers
10 views

Should the partials be averaged once or multiple times for backprop

I am trying to make a general purpose symbolic differentiation library, and I am not sure how to handle mini batch learning for SGD. When back-propagating, the sums of the partials are usually added ...
1
vote
1answer
17 views

Backpropgating error to emedding matrix

I understand the backpropagation algorithm of neural networks, and how the error propagates backwards in layers. That is, I understand that given a 3-layer feed forward network, the amount to change ...
4
votes
1answer
104 views

Backpropagation - softmax derivative

I have a question on the backpropagation in a simple neural network (I am trying to derive the derivative for the backpropagation). Suppose that the network is simple like so (forward pass): $$\...
2
votes
0answers
43 views

(Deep Learning) Backpropagation derivation from notes by Andrew NG

I am self-studying Andrew NG's deep learning course materials from the mcahine learning course (CS 229) of Stanford. The material is available here: http://cs229.stanford.edu/notes/cs229-notes-...
0
votes
0answers
36 views

Pytorch: Combining Automatic and Manual Methods

I am testing a two-step architecture that is composed of a conventional first section that can be implemented with any standard deep learning architecture and a second section that must be coded ...
1
vote
0answers
67 views

Alternative method for RNN backpropagation through time

I am experimenting with a character-level LSTM model doing the standard task of predicting the next character given a sequence of characters. I am training the network using truncated backpropagation ...
4
votes
2answers
144 views

Synthetic Gradients - doesn't seem beneficial

I can see two motives to use Synthetic Gradients in RNN: To speed up training, by imediately correcting each layer with predicted gradient To be able to learn longer sequences I see ...
1
vote
1answer
86 views

Could someone explain to me how back-prop is done for the generator in a GAN?

I'm not very familiar with neural networks, however, I though I understood the concept of back propagation as starting from the error in the output layer. Say, we have 3 neurons in the output layer ...
3
votes
0answers
67 views

Synthetic Gradients good number of Layers & neurons

I would like to train my LSTM with a "synthetic gradients" Decoupled Neural Interface (DNI). How to decide on the number of layers and neurons for my DNI? Searching them by trial end error or ...
-1
votes
1answer
33 views

How to understand backpropagation using derivative [duplicate]

Before that I was learning about gradient descent and I understand this, it's ok for me. Now I have a problem with backpropagation algorithm. I know the idea - minimalize error in multilayer neural ...
3
votes
0answers
32 views

Hochreiter LSTM (p. 4): Maximal values of logistic sigmoid derivative times weight

My questions follow the below page 4 excerpt from Hochreiter's LSTM paper: If $f_{l_{m}}$ is the logistic sigmoid function, then the maximal value of $f^\prime_{l_{m}}$ is 0.25. If $y^{l_{m-1}}$ ...
1
vote
2answers
48 views

neural network xor gate classification

I've written a simple neural network that can predict XOR gate function. I think I've used the math correctly, but the loss doesn't go down and remains near 0.6. Can anyone help me find the reason why?...
0
votes
0answers
21 views

Big Difference in Training from different output scaling methods

I am training an AC model on Pendulum-v0. After some very, very frustrating weekends and lots of gridsearch on params, I have found that one place in particular has made all of the difference. This ...
0
votes
1answer
22 views

Can you sum up gradients and apply in batch?

I'm following this Reinforcement Learning Tutorial. There, training data is collected during an episode. When the episode is done, the data is used to do backpropagation. However, instead of applying ...
2
votes
1answer
37 views

GradientChecking, can I blame float precision?

I am trying to GradientCheck my c++ LSTM. The structure is as follows: output vector (5D) Dense Layer with softmax (5 neurons) LSTM layer (5 neurons) input vector (5D) My gradient ...
3
votes
1answer
72 views

Why does momentum need learning rate?

If momentum optimizer independently keeps a custom "inertia" value for each weight, then why do we ever need to bother with learning rate? Surely, momentum would catch up its magnutude pretty ...
1
vote
2answers
41 views

How does backpropagation differ from reverse-mode autodiff

Going through this book, I am familiar with the following: For each training instance the backpropagation algorithm first makes a prediction (forward pass), measures the error, then goes through ...
4
votes
2answers
69 views

deriving the gradient of batch normalization

I'm trying to figure out the gradient of batch norm wrt x for backprop, but I get stuck in what I will call 'the triangle of (gradient) death'. I present to you the triangle of death (in red), in the ...
1
vote
0answers
21 views

Should there be 'total derivative' symbol in the mathematical representation of back-propagation algorithm's formula?

I curiously want to know why in many references [1][2] on back propagation, authors use the partial derivative symbol, i.e $\frac{\partial}{\partial W}$, instead of correctly alternating between it ...
0
votes
0answers
31 views

Self driving car using LSTM does not learn to turn. Stuck at local minima?

I am trying to teach a self driving car to drive using a LSTM. The first part I have trouble with is to make the car drive between the lanes. As input to the LSTM I use the lanes ahead of the car. I ...
4
votes
1answer
351 views

How to apply the gradient of softmax in backprop

I recently did a homework where I had to learn a model for the MNIST 10-digit classification. The HW had some scaffolding code and I was supposed to work in the context of this code. My homework ...
0
votes
0answers
7 views

Multiply by parent gradient at the very end?

I have a parent-gradient flowing into the activations of my final layer in my net (call this gradient $G$). Is it possible to first compute the gradient for all the preceding layers of my net, and ...
1
vote
2answers
50 views

Homemade deep learning library: numerical issue with relu activation

For the sake of learning the finer details of a deep learning neural network, I have coded my own library with everything (optimizer, layers, activations, cost function) homemade. It seems to work ...
1
vote
2answers
60 views

is Batch Norm a little bit stochastic by default?

Using full-batch gradient descent, stacking 100 layers and using alpha 0.0001 results in steadily decreasing error. However, after I implemented Batch Norm, the same scenario results in fluctuations. ...
1
vote
0answers
137 views
1
vote
2answers
112 views

A good reference for the back propagation algorithm?

I'm trying to learn more about the fundamentals of neural networks. I feel like I understand the basics of back propagation, but I want to solidify the details in my mind. I was working through Ian ...
0
votes
0answers
14 views

modified rmsprop with ^4 instead of ^2

The rmsprop optimization algorithm looks as follows: $$dw = \frac{\partial E}{\partial w}$$ $$S_{dw} = (1-\beta)S_{dw} + \beta \left(dw \right) ^2$$ $$W = W - \alpha \frac{dw}{\sqrt{S_{dw}}}$$ What ...
1
vote
0answers
12 views

I Can't find my RMSProp implementation bug?

I'm trying to implement RMSProp in my own Neural Network library so I can undertand the 'under-the-hood' operations, but this specific implementation is not working / converging, and I can't figure ...
4
votes
1answer
219 views

How to propagate error back to previous layer in CNN?

I have a convolutional layer (link) with an input 5x5x2 (width, height, depth): The layer has 3 filters with dimensions 3x3x2, it produces an output with dimensions 3x3x3. I have completed the ...