Questions tagged [backpropagation]

Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Backpropagation: Relevance of the error signal of a neuron

During my quest to understand back propagation in a more rigorous approach I have come across with the definition of error signal of a neuron which is defined as follows for the $j^{\text{th}}$ neuron ...
1
vote
1answer
24 views

Backpropagation with log likelihood cost function and softmax activation

In the online book on neural networks by Michael Nielsen, in chapter 3, he introduces a new cost function called as log-likelihood function defined as below $$ C = -ln(a_y^L) $$ Suppose we have 10 ...
0
votes
0answers
7 views

Effect of ReLu derivative in convolution layer backpropagation

I'm trying to implement a CNN, as part of an academic project to learn how it works. The project is a SRCNN: a convolutional neural network that increase the resolution of images. Following this ...
1
vote
0answers
12 views

How to backpropogate error from convolutional layer with respect to the input when using multiple channels

I have been attempting to implement a Convolutional Neural Network in python and have run into a bit of a roadblock. When backpropogating the error in a convolutional layer let us say that we receive ...
1
vote
0answers
8 views

How does the Backpropagation through time work?

I have to write a paper on LSTMs and I want to explain why LSTMs exist in the first place. According to some papers and books because usual RNNs had problems with vanishing gradients and the LSTM has ...
0
votes
0answers
17 views

Forward and backward pass in Conv2D transpose Layer

I’ve several questions regarding the transposed convolution 2d layer. I’ve not been able to find a proper resource explaining the forward and backward pass. What I know (but not for sure) is, that ...
0
votes
1answer
12 views

BPTT vs Vanishing Gradient Problem

I know that BPTT is the method to apply Back Propagation on RNN. Which is works fine with RNN as it stops at certain point as changes approach to zero but isn't it the exact Vanishing Gradient ...
0
votes
0answers
22 views

Backpropagation Max-Pooling Layers: Multiple Maximum Values

I am currently implementing a CNN in plain numpy and have a brief question regarding a special case of the backpropagation for a max-pool layer: While it is clear that the gradient with respect to ...
1
vote
1answer
40 views

Backpropagation in a convolutional neural network with stride and padding

So i am trying to learn backpropagation of convolutional neural networks. A lot of articles only cover convolutions without a stride and a padding variable, so i decided to try it on my own. For ...
2
votes
0answers
11 views

Computing derivatives for backpropagation across a convolution step

This will be a long post, but I hope it'll be instructive to anyone else in my position. I'm trying to find how the derivatives of the loss function are calculated with respect to the kernels and ...
1
vote
1answer
19 views

Reason behind the sum of rate factors for calculating cost function derivative

Suppose we have a network of neurons like below: We make a little change in weight w[l][j][k] on our network, and it can make change on our cost function from ...
0
votes
0answers
4 views

What are the main reasons which does not cause the training error of yolov2 to not diminish?

I am using https://github.com/thtrieu/darkflow yolov2 for detecting and classifying the images of passport. There are 8 classes and all the objects are passport details like name, father name, mother ...
0
votes
0answers
13 views

Does a max-pooling layer in a ConvNet contribute to the “vanishing gradient” problem?

I would answer no, but am not sure if I'm missing something and hope you can help me out: The derivative of a max-pooling layer in a ConvNet is one w.r.t. the maximum value and zero for all others. A ...
0
votes
0answers
10 views

Chaining bias terms in backprob

let's say I have a few linear layers $l_1 \dots l_n$: $y=I(\dots I(IX + b_1) + b_2) \dots +b_n)$ where $n$ is sufficiently large and $I$ is the (nonparametric) identity matrix. The gradient for $b_n$...
0
votes
2answers
14 views

Is the Cross Entropy Loss important at all, because at Backpropagation only the Softmax probability and the one hot vector are relevant?

Is the Cross Entropy Loss (CEL) important at all, because at Backpropagation (BP) only the Softmax (SM) probability and the one hot vector are relevant? When applying BP, the derivative of CEL is the ...
1
vote
1answer
38 views

Neural network is getting partially trained

So I am writing my own neural network library using back-propagation as my training algorithm. Everything seems fine the error is getting decreased more and more at each iteration however when I am ...
0
votes
1answer
40 views

Max pooling has no parameters and therefore doesn't affect the backpropagation?

I feel this is a question that has a lot of variations already posted but it doesn't exactly answer my question. I understand the concept of max pooling and also the concept of backpropagation. What i ...
0
votes
1answer
14 views

How do I convert a summation equation to a vector equation (backpropagation)?

$$a_j^l=\sigma(\sum_{k} {w_j}_k^l {a}_k^{l-1}+b_j^l)$$ $$a^l=\sigma( w^l {a}^{l-1}+b^l)$$ In a resource I have been reading, the above equations describe the activation of a neurone. They have the ...
0
votes
1answer
18 views

Confused with the derivation of the gradient descent update rule

I have been going over some theory for gradient descent. The source I am looking at said that the change in cost can be described by the following equation: $$∆C=∇C∙∆w$$ where $∇C$ is the gradient ...
5
votes
0answers
70 views

Differences between gradient calculated by different reduction methods in PyTorch

I'm playing with different reduction methods provided in built-in loss functions. In particular, I would like to compare the following. The averaged gradient by performing backward pass for each loss ...
2
votes
1answer
45 views

Layer notation for feed forward neural networks

Apologies in advance, for I have a fairly rudimentary question on the notations for studying Feed-Forward Neural Networks. Here is a nice schematic taken from this blog-post. Here $x_i = f_i(W_i \...
0
votes
1answer
17 views

What does this expression from gradient descent mean?

I am looking over some neural network theory and came across this equation, coupled with this description (gradient descent ball-valley analogy): ''let's think about what happens when we move the ...
1
vote
0answers
47 views

Having problem in back propagation part for dimension

I was trying to build a neural network with single hidden layer from scratch. In back propagation part some problems have raised. For calculating gradient of loss function with respect to weight in ...
1
vote
1answer
38 views

LeCun paper on deeplearning (Nature, 2015)

As I was reading Y. LeCun's paper on Deep Learning (Nature, vol. 521, 2015), I came across a figure (the 1st one in the paper) which was associated to the backward ...
0
votes
1answer
22 views

Back propagation on matrix of weights

I am trying to implement a Neural Network for binary classification using python and numpy only. My network structure is as follows: input features: 2 [1X2] matrix Hidden layer1: 5 neurons [2X5] ...
2
votes
1answer
69 views

Dueling Network gradient with respect to Advantage stream

Looking at Dueling DQN: $Q = V + A - mean(A)$ For simplicity, let's assume we are working with 4 neurons. Recall that Value stream only has 1 neuron $(v_0)$ Re-writing the above equation, we get: $...
0
votes
4answers
40 views

Is backpropogation used in convolutional neural networks?

Do convolutional neural network use the backpropogation algorithm? I am not understanding what exactly happens in fully connected layers?
2
votes
0answers
19 views

Get derivatives from your NN

How can I get the gradient of a node in the NN with respect to another one? I need to train a NN, which for the sake of simplicity has 2 neurons as input (x, y), a neuron as a bottleneck (z), and 2 ...
1
vote
0answers
41 views

CartPole v1 - Simple backprop with 1 hidden layer

I'm trying to solve the CartPole-v1 problem from OpenAI by using backprop on a one-layer neural network - while updating the model at every time step using State action values (Q(s,a)). I'm unable to ...
0
votes
0answers
15 views

Layer-wise relevance propagation for the input layer

I'm studying this argoument and it looks pretty clear to me. I was looking to the LRP $\alpha_1\beta_0$ implementation, that's equal to the Deep-Taylor decomposition. The idea is the following, start ...
0
votes
0answers
21 views

Implementation of Layer-wise relevant propagation in MATLAB from scratch

I'm having serious issues with the implementation of the LRP algorithm for neural networks in MATLAB. The challenge is to implement the equations correctly. I'm trying to implement the deep-Taylor $\...
2
votes
1answer
129 views

Derivation of backpropagation for Softmax

So, after a couple dozen tries I finally implemented a standalone nice and flashy softmax layer for my neural network in numpy. All works well, but I have a question regarding the maths part because ...
0
votes
0answers
15 views

Is this an incorrect way of back-propagating error with matrices?

I was watching a public available video from Stanford (https://youtu.be/d14TUNcbn1k?t=2720) on the mathematics behind back propagation. They proposed a graph: that was then used as an example of back ...
0
votes
0answers
61 views

How to handle maxpool layer backpropagation with recurring max values in same position

Say I have a layer a: 3 4 2 1 5 0 8 6 4 The maxpool using 2x2 filter is: <...
0
votes
0answers
13 views

Backpropagation implementation help

I'm trying to implement Nokland's Direct Feedback Alignment in Python following his paper. Here's my implementation so far: ...
0
votes
1answer
19 views

What are the criteria for updating bias values in back propagation?

During back propagation, the algorithm can modify the weight values or bias values to reduce the loss. How does the algorithm decide whether it has to modify the weight values or bias values to ...
0
votes
0answers
18 views

My question is about dependency between hidden states for Back Propagation Through Time in RNN

In one video lecture, professor Ali Ghodsi of University of Waterloo says that the first node of S(t)(hidden state of RNN at time t) has an effect only on the first node of S(t+1)(hidden state of RNN ...
3
votes
2answers
163 views

CNN Back Propagation without Sigmoid Derivative

I'm new to CNN and trying to study some MATLAB sample codes (cause I need to know the internal calculation). I recently realized that the sample code I'm using doesn't multiply error by sigmoid's ...
0
votes
1answer
42 views

How does back propagation works through layers like maxpooling and padding? [duplicate]

I know back propagation takes derivatives (changing one quantity wrt other). But how this is applied when there is maxpooling layer in between two Conv2D layers? How it gains its original shape when ...
0
votes
0answers
13 views

Would traditional ML approaches benefit from a feedback loop (a la backprop)?

Would classical ML approaches such as SVMs, RF etc. benefit by having some sort of feedback mechanism (similar to backpropagation in Deep learning)? If so, how can one go about it? Is there any ...
1
vote
1answer
60 views

Backpropagation for noobs

I am trying to understand neural networks and how they work, by programming my own one from scratch in nodejs. Currently, i managed to build a network, that has weights, layers and neurons. I also ...
5
votes
1answer
1k views

What do “compile” , “fit” and “predict” do in Keras sequential models?

I am a little confused between these two parts of Keras sequential models functions. May someone explains what is exactly the job of each one? I mean ...
2
votes
0answers
215 views

Pytorch: How to create an update rule the doesn't come from derivatives?

I want to implement the following algorithm, taken from this book, section 13.6: Here, the neural networks' outputs are $V(S, w)$ and $\pi(A|S,\theta)$, parameterized by $w$ and $\theta$ respectively....
1
vote
1answer
18 views

quest for online tutorials on CNN backprop

I'm looking for video or text tutorial materials on CNN backprop that show how weights (convolutional kernels and last layer weights) are trained. Anyone know?
3
votes
1answer
327 views

How does Gradient Descent and Backpropagation work together?

Please forgive me as I am new to this. I have attached a diagram trying to model my understanding of neural network and Back-propagation? From videos on coursera and resources online I formed the ...
1
vote
0answers
45 views

Backpropagation using only numpy

I've been trying to implement the backpropagation algorithm using only numpy, I've already done the Keras version, but when implementing the numpy version, the loss is diverging as seen in the image ...
1
vote
1answer
47 views

Understanding backprop for softmax

I'm looking on a given solution of the first assignment of cs231n course. Down below a snippet from the loss function. I don't really understand lines 140-143. Can ...
0
votes
1answer
41 views

Information about LTSM RNN backpropagation algorithm

I am attempting to make a LTSM RNN in python from scratch and I have completed the code for forward pass but I am struggling to find a clear outline of the equations I need to calculate to get the ...
0
votes
1answer
46 views

What is the difference between reconstruction vs backpropagation?

I was following a tutorial on understanding Restricted Boltzmann Machines (RBMs) and I noticed that they used both the terms reconstruction and ...
1
vote
0answers
41 views

Using machine learning algorithm to approximate a matrix? [closed]

I don't know if this is a good question. I would like to use machine learning to approximate a matrix. For example, a very simple one as \begin{align} A=\begin{pmatrix} 1 &0 \\ 0& 2 \end{...