A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [backpropagation]

Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
2
votes
1answer
23 views

Interpreting Gradients and Partial Derivatives when training Neural Networks

I am trying to understand of purpose of partial differentiation in NN training by knowing how to interpret gradients and their partial derivatives. Below is my way of interpreting them so I would like ...
1
vote
2answers
18 views

Justification for values used in backpropagation

I'm learning the method for backpropagation in adjusting weights. A generalization of a formula used to determine the change made to a respective weight is where is the rate the total error changes ...
1
vote
1answer
28 views

Implementing “full convolution” to find gradient w.r.t the convolution layer inputs

I've been trying to implement "full convolution" w.r.t to convolution layer inputs. According to this article, it looks like this: So, I wrote this function: ...
2
votes
2answers
36 views

Forward pass vs backward pass vs backpropagation

As mentioned in the question, i have some issues understanding what are the differences between those terms. From what i have understood: 1) Forward pass: compute the output of the network given the ...
2
votes
0answers
24 views

backpropagation between fully connected layer and convolution layer?

This is a simple example of a network consisting of two convolutional layers and one fully connected layer. ...
0
votes
0answers
13 views

Backpropagation through the inputs of convolutional layer in LeNet

I am trying to understand backprop for LeNet according to the original article http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf I think I have successfully done backdrop till the C3 ...
0
votes
1answer
29 views

Why multiply by 2 when calculating partial derivatives during backpropagation?

I'm wondering why we multiple by 2 when calculating partial derivatives. I'm referencing the 2's that I've circled below, from here. We also see this in the python implementation, ...
3
votes
2answers
52 views

Hello, anyone able to direct me to a “cheat sheet” of Neural Network equations with legends?

Can anyone here can direct me to a site that provides a cheat sheet of equations for Neural Networks with a legend for the notation? It can be on any and all aspects of NN, be it forward or back ...
1
vote
0answers
10 views

Question regarding: Vectorization Math of Backpropagation in a Neural Network

Formula: These are the formula I use for backpropagation from Brilliant: Question: If we consider a Neural Network with the structure (3,2): And we would start calculating the derivative (for 1 ...
2
votes
1answer
34 views

Problem with chain rule in softmax layer when differentiated separately

I have some problems with backpropagation in softmax output layer. I know how it should work but if I try to apply the chain rule in the classical way, I get different results compared to when Softmax ...
1
vote
0answers
22 views

How to do Back Propagation Updation for below code?

Below is the code by Siraj Raval for implementing Neural Networks from Scratch. I have some doubts regarding the Code: Why during updation he did W2 = W2 + L1.T.dot(L2_Delta). I Mean shouldn't it be ...
1
vote
0answers
33 views

Covariance-Matrix Adaptation Evolution Strategy (CMA-ES) implementation with Tensorflow Sequential model

After coming across this article about evolution strategies http://blog.otoro.net/2017/10/29/visual-evolution-strategies/, it seems clear that the Covariance-Matrix Adaptation Evolution Strategy (CMA-...
1
vote
0answers
106 views

Neural Network from scratch: cost increasing over epochs

I'm trying to design a neural network from scratch. After training my neural network, I make a plot of the cost vs epochs, which I would expect to decrease throughout the runtime of the NN training, ...
0
votes
0answers
15 views

Backprop: backward pass way faster than forward pass

I started to work with my own implementation of backpropagation algorithm, that I made five years ago. For each training sample (input-output pair), I make a forward pass (to compute outputs of each ...
1
vote
1answer
34 views

Doubt in Derivation of Backpropagation

I was going through the derivation of backpropagation algorithm provided in this document (adding just for reference). I have doubt at one specific point in this derivation. The derivation goes as ...
0
votes
0answers
15 views

What's the input for the cost function?

I'm trying to implement deep Q-learning, but I do not know what to put into the cost function. My net has 8 scalar inputs, 4 scalar outputs (from 0-1) and no hidden layers. To calculate the cost I ...
0
votes
1answer
73 views

Back-propagation and stochastic gradient descent

Is backpropagation a learning method or an optimisation method? How are backpropagation and stochastic gradient descent related to each other?
1
vote
1answer
26 views

Backpropagation chain rule example

My question is in regards to an MIT course example. The instructor delves into the backpropagation of this simple NN. I have two questions. Why do we seem to disregard the weights of the second ...
0
votes
0answers
10 views

MADALINE learning vs modern backpropagation

I am just a tad bit confused from my reading. If we have a multiple layer ADALINE NN (MADALINE) (or perceptron even); how would this have been trained prior to backpropagation? If I am correct, with ...
0
votes
0answers
28 views

How to derive gradients for softmax function

We have the following feedforward equations: $z_1 = W_1x + b_1$ $a_1 = f(z_1)$ $z_2 = W_2a_1 + b_2$ $a_2 = y^* = softmax(z_2)$ $L(y, y^*) = -\frac{1}{N}\sum_{n \in N} \sum_{i \in C} y_{n,i} \log{...
0
votes
0answers
25 views

binary cross entropy loss - pixels wise predictions - How error is back propagated?

I relate for example in a semantic segmentation task of picture 100 by 100 pixels. The task is to segment the picture using a mask for each picture as the label. Mask has values 0 and 1. (re. Unet). I ...
0
votes
0answers
30 views

Backpropagation incorrect only for hidden layer bias node

I am trying to implement a neural network in Python, however I am having some trouble when it comes to the implementation of backpropagation. I have been checking my results with this example and ...
0
votes
1answer
48 views

How CNN applies backpropagation to update its weights and biases?

I understand that the 3 main layers for CNN are convolutional layer, ReLU layer and pooling layer. However, I do not understand how CNN updates its weights and biases using backpropagation. I ...
0
votes
0answers
17 views

Can I update weights of keras neural net only if validation improves?

I am training a neural network in keras and I reach a classical limit - my training accuracy improves with increasing epochs, but my validation accuracy decreases after 9 epochs (see figure). I ...
0
votes
1answer
538 views

LSTM loss function and backpropagation

I'm trying to understand the connection between loss function and backpropagation. From what I understood until now, backpropagation is used to get and update matrices and bias used in forward ...
3
votes
4answers
112 views

Backpropagation: Relevance of the error signal of a neuron

During my quest to understand back propagation in a more rigorous approach I have come across with the definition of error signal of a neuron which is defined as follows for the $j^{\text{th}}$ neuron ...
1
vote
1answer
111 views

Backpropagation with log likelihood cost function and softmax activation

In the online book on neural networks by Michael Nielsen, in chapter 3, he introduces a new cost function called as log-likelihood function defined as below $$ C = -ln(a_y^L) $$ Suppose we have 10 ...
0
votes
0answers
18 views

Effect of ReLu derivative in convolution layer backpropagation

I'm trying to implement a CNN, as part of an academic project to learn how it works. The project is a SRCNN: a convolutional neural network that increase the resolution of images. Following this ...
1
vote
0answers
21 views

How to backpropogate error from convolutional layer with respect to the input when using multiple channels

I have been attempting to implement a Convolutional Neural Network in python and have run into a bit of a roadblock. When backpropogating the error in a convolutional layer let us say that we receive ...
1
vote
0answers
15 views

How does the Backpropagation through time work?

I have to write a paper on LSTMs and I want to explain why LSTMs exist in the first place. According to some papers and books because usual RNNs had problems with vanishing gradients and the LSTM has ...
0
votes
0answers
39 views

Forward and backward pass in Conv2D transpose Layer

I’ve several questions regarding the transposed convolution 2d layer. I’ve not been able to find a proper resource explaining the forward and backward pass. What I know (but not for sure) is, that ...
1
vote
2answers
28 views

BPTT vs Vanishing Gradient Problem

I know that BPTT is the method to apply Back Propagation on RNN. Which is works fine with RNN as it stops at certain point as changes approach to zero but isn't it the exact Vanishing Gradient ...
0
votes
0answers
67 views

Backpropagation Max-Pooling Layers: Multiple Maximum Values

I am currently implementing a CNN in plain numpy and have a brief question regarding a special case of the backpropagation for a max-pool layer: While it is clear that the gradient with respect to ...
1
vote
1answer
182 views

Backpropagation in a convolutional neural network with stride and padding

So i am trying to learn backpropagation of convolutional neural networks. A lot of articles only cover convolutions without a stride and a padding variable, so i decided to try it on my own. For ...
2
votes
0answers
22 views

Computing derivatives for backpropagation across a convolution step

This will be a long post, but I hope it'll be instructive to anyone else in my position. I'm trying to find how the derivatives of the loss function are calculated with respect to the kernels and ...
1
vote
1answer
27 views

Reason behind the sum of rate factors for calculating cost function derivative

Suppose we have a network of neurons like below: We make a little change in weight w[l][j][k] on our network, and it can make change on our cost function from ...
0
votes
0answers
21 views

What are the main reasons which does not cause the training error of yolov2 to not diminish?

I am using https://github.com/thtrieu/darkflow yolov2 for detecting and classifying the images of passport. There are 8 classes and all the objects are passport details like name, father name, mother ...
0
votes
0answers
34 views

Does a max-pooling layer in a ConvNet contribute to the “vanishing gradient” problem?

I would answer no, but am not sure if I'm missing something and hope you can help me out: The derivative of a max-pooling layer in a ConvNet is one w.r.t. the maximum value and zero for all others. A ...
0
votes
0answers
14 views

Chaining bias terms in backprob

let's say I have a few linear layers $l_1 \dots l_n$: $y=I(\dots I(IX + b_1) + b_2) \dots +b_n)$ where $n$ is sufficiently large and $I$ is the (nonparametric) identity matrix. The gradient for $b_n$...
0
votes
2answers
29 views

Is the Cross Entropy Loss important at all, because at Backpropagation only the Softmax probability and the one hot vector are relevant?

Is the Cross Entropy Loss (CEL) important at all, because at Backpropagation (BP) only the Softmax (SM) probability and the one hot vector are relevant? When applying BP, the derivative of CEL is the ...
1
vote
1answer
45 views

Neural network is getting partially trained

So I am writing my own neural network library using back-propagation as my training algorithm. Everything seems fine the error is getting decreased more and more at each iteration however when I am ...
0
votes
1answer
131 views

Max pooling has no parameters and therefore doesn't affect the backpropagation?

I feel this is a question that has a lot of variations already posted but it doesn't exactly answer my question. I understand the concept of max pooling and also the concept of backpropagation. What i ...
0
votes
1answer
15 views

How do I convert a summation equation to a vector equation (backpropagation)?

$$a_j^l=\sigma(\sum_{k} {w_j}_k^l {a}_k^{l-1}+b_j^l)$$ $$a^l=\sigma( w^l {a}^{l-1}+b^l)$$ In a resource I have been reading, the above equations describe the activation of a neurone. They have the ...
0
votes
1answer
39 views

Confused with the derivation of the gradient descent update rule

I have been going over some theory for gradient descent. The source I am looking at said that the change in cost can be described by the following equation: $$∆C=∇C∙∆w$$ where $∇C$ is the gradient ...
8
votes
1answer
500 views

Differences between gradient calculated by different reduction methods in PyTorch

I'm playing with different reduction methods provided in built-in loss functions. In particular, I would like to compare the following. The averaged gradient by performing backward pass for each loss ...
2
votes
1answer
81 views

Layer notation for feed forward neural networks

Apologies in advance, for I have a fairly rudimentary question on the notations for studying Feed-Forward Neural Networks. Here is a nice schematic taken from this blog-post. Here $x_i = f_i(W_i \...
0
votes
1answer
21 views

What does this expression from gradient descent mean?

I am looking over some neural network theory and came across this equation, coupled with this description (gradient descent ball-valley analogy): ''let's think about what happens when we move the ...
1
vote
0answers
62 views

Having problem in back propagation part for dimension

I was trying to build a neural network with single hidden layer from scratch. In back propagation part some problems have raised. For calculating gradient of loss function with respect to weight in ...
1
vote
1answer
61 views

LeCun paper on deeplearning (Nature, 2015)

As I was reading Y. LeCun's paper on Deep Learning (Nature, vol. 521, 2015), I came across a figure (the 1st one in the paper) which was associated to the backward ...
0
votes
1answer
33 views

Back propagation on matrix of weights

I am trying to implement a Neural Network for binary classification using python and numpy only. My network structure is as follows: input features: 2 [1X2] matrix Hidden layer1: 5 neurons [2X5] ...