Questions tagged [backpropagation]

Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Backpropagation through time details clarification

The way I understand back-propagation in time could be implemented in the following way: Go through the provided sequence, store the resulting hidden states of the network Iterate through the ...
1
vote
0answers
19 views

If mean absolute loss is not differentiable, how it can be used in neural networks? which majorly are trained using back-propagation

If Mean Absolute Error (MAE) loss is not differentiable, how can it be used in neural networks? which majorly are trained using back-propagation I am wondering if MAE is not differentiable how they ...
0
votes
1answer
25 views

Generator losses in WGAN and potential convergence failure

I have been training a WGAN for a while now, with my generator training once in every five epochs. I have tried several model architectures(no of filters) and also tried varying the relationship with ...
0
votes
0answers
36 views

How is the error function constructed for backpropagation?

We know that the backpropagation works in the way that the derivative is calculated from the error function. But how is the error function constructed? 1) Is it so that at first there is only one ...
0
votes
0answers
22 views

Backpropagation LSTM: compute gradient over multiple timesteps

Has anyone implemented simplified code to compute gradient of error over multiple timesteps for a single example. If the timesteps are large, even solving on paper it is getting really complicated. ...
2
votes
1answer
37 views

How many times is backprop used in epoch?

As I understand for the algorithms that use gradient descent we have to pass data to the algorithms multiple times so that the optimum is found. So one epoch means that the forward-backprop (and ...
0
votes
0answers
27 views

Custom training loss with custom gradients

I am trying to write a custom loss in Tensorflow v2, for simplicity let's say that I'm using Mean Squared Error loss as follows, ...
0
votes
2answers
23 views

Auto-Encoder customized layer training

My question is related with model-weights optimization during back propagation. In this image I'm trying to represent an auto-encoder having 7 layers where 4th one is center layer. If my ...
1
vote
0answers
10 views

Should the weights be rotated when using SciPy full convolution?

I use SciPy's single.convolve2d in "full" mode to compute gradient w.r.t to convolution layer inputs. In my current implementation, I don't rotate filters as suggested by this article because I assume ...
2
votes
0answers
24 views

LONG SHORT-TERM MEMORY Hochreiter's paper BPTT

I was trying to read paper about LSTM, and I am stuck with mathematical problem. http://www.bioinf.jku.at/publications/older/2604.pdf page 4. see, |$f'_l$$_m$($net_l$$_m$)$w_l$$_m$ $_l$$_m$$_-$$_1$...
1
vote
1answer
23 views

What is the layer above/below in a NN?

In the lecture notes of CS231n, it says (emphasis mine) ... There are three major sources of memory to keep track of: From the intermediate volume sizes: These are the raw number of ...
0
votes
0answers
25 views

Manual Calculation for Backpropagation

I am trying to understand the backpropogation in neural net, Take a look at this below picture Here, I understand the formula for calculating error in Weight5(w5) and also updating the w5. But, I am ...
2
votes
1answer
77 views

Interpreting Gradients and Partial Derivatives when training Neural Networks

I am trying to understand of purpose of partial differentiation in NN training by knowing how to interpret gradients and their partial derivatives. Below is my way of interpreting them so I would like ...
1
vote
2answers
24 views

Justification for values used in backpropagation

I'm learning the method for backpropagation in adjusting weights. A generalization of a formula used to determine the change made to a respective weight is where is the rate the total error changes ...
1
vote
1answer
35 views

Implementing “full convolution” to find gradient w.r.t the convolution layer inputs

I've been trying to implement "full convolution" w.r.t to convolution layer inputs. According to this article, it looks like this: So, I wrote this function: ...
2
votes
2answers
311 views

Forward pass vs backward pass vs backpropagation

As mentioned in the question, i have some issues understanding what are the differences between those terms. From what i have understood: 1) Forward pass: compute the output of the network given the ...
2
votes
0answers
27 views

backpropagation between fully connected layer and convolution layer?

This is a simple example of a network consisting of two convolutional layers and one fully connected layer. ...
0
votes
0answers
15 views

Backpropagation through the inputs of convolutional layer in LeNet

I am trying to understand backprop for LeNet according to the original article http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf I think I have successfully done backdrop till the C3 ...
0
votes
1answer
31 views

Why multiply by 2 when calculating partial derivatives during backpropagation?

I'm wondering why we multiple by 2 when calculating partial derivatives. I'm referencing the 2's that I've circled below, from here. We also see this in the python implementation, ...
2
votes
2answers
57 views

Hello, anyone able to direct me to a “cheat sheet” of Neural Network equations with legends?

Can anyone here can direct me to a site that provides a cheat sheet of equations for Neural Networks with a legend for the notation? It can be on any and all aspects of NN, be it forward or back ...
1
vote
0answers
12 views

Question regarding: Vectorization Math of Backpropagation in a Neural Network

Formula: These are the formula I use for backpropagation from Brilliant: Question: If we consider a Neural Network with the structure (3,2): And we would start calculating the derivative (for 1 ...
2
votes
1answer
44 views

Problem with chain rule in softmax layer when differentiated separately

I have some problems with backpropagation in softmax output layer. I know how it should work but if I try to apply the chain rule in the classical way, I get different results compared to when Softmax ...
1
vote
0answers
22 views

How to do Back Propagation Updation for below code?

Below is the code by Siraj Raval for implementing Neural Networks from Scratch. I have some doubts regarding the Code: Why during updation he did W2 = W2 + L1.T.dot(L2_Delta). I Mean shouldn't it be ...
1
vote
0answers
80 views

Covariance-Matrix Adaptation Evolution Strategy (CMA-ES) implementation with Tensorflow Sequential model

After coming across this article about evolution strategies http://blog.otoro.net/2017/10/29/visual-evolution-strategies/, it seems clear that the Covariance-Matrix Adaptation Evolution Strategy (CMA-...
1
vote
0answers
106 views

Neural Network from scratch: cost increasing over epochs

I'm trying to design a neural network from scratch. After training my neural network, I make a plot of the cost vs epochs, which I would expect to decrease throughout the runtime of the NN training, ...
0
votes
0answers
23 views

Backprop: backward pass way faster than forward pass

I started to work with my own implementation of backpropagation algorithm, that I made five years ago. For each training sample (input-output pair), I make a forward pass (to compute outputs of each ...
1
vote
1answer
35 views

Doubt in Derivation of Backpropagation

I was going through the derivation of backpropagation algorithm provided in this document (adding just for reference). I have doubt at one specific point in this derivation. The derivation goes as ...
0
votes
0answers
15 views

What's the input for the cost function?

I'm trying to implement deep Q-learning, but I do not know what to put into the cost function. My net has 8 scalar inputs, 4 scalar outputs (from 0-1) and no hidden layers. To calculate the cost I ...
0
votes
1answer
139 views

Back-propagation and stochastic gradient descent

Is backpropagation a learning method or an optimisation method? How are backpropagation and stochastic gradient descent related to each other?
1
vote
1answer
29 views

Backpropagation chain rule example

My question is in regards to an MIT course example. The instructor delves into the backpropagation of this simple NN. I have two questions. Why do we seem to disregard the weights of the second ...
0
votes
0answers
17 views

MADALINE learning vs modern backpropagation

I am just a tad bit confused from my reading. If we have a multiple layer ADALINE NN (MADALINE) (or perceptron even); how would this have been trained prior to backpropagation? If I am correct, with ...
0
votes
0answers
52 views

How to derive gradients for softmax function

We have the following feedforward equations: $z_1 = W_1x + b_1$ $a_1 = f(z_1)$ $z_2 = W_2a_1 + b_2$ $a_2 = y^* = softmax(z_2)$ $L(y, y^*) = -\frac{1}{N}\sum_{n \in N} \sum_{i \in C} y_{n,i} \log{...
0
votes
0answers
65 views

binary cross entropy loss - pixels wise predictions - How error is back propagated?

I relate for example in a semantic segmentation task of picture 100 by 100 pixels. The task is to segment the picture using a mask for each picture as the label. Mask has values 0 and 1. (re. Unet). I ...
0
votes
0answers
62 views

Backpropagation incorrect only for hidden layer bias node

I am trying to implement a neural network in Python, however I am having some trouble when it comes to the implementation of backpropagation. I have been checking my results with this example and ...
0
votes
1answer
80 views

How CNN applies backpropagation to update its weights and biases?

I understand that the 3 main layers for CNN are convolutional layer, ReLU layer and pooling layer. However, I do not understand how CNN updates its weights and biases using backpropagation. I ...
0
votes
0answers
21 views

Can I update weights of keras neural net only if validation improves?

I am training a neural network in keras and I reach a classical limit - my training accuracy improves with increasing epochs, but my validation accuracy decreases after 9 epochs (see figure). I ...
0
votes
1answer
1k views

LSTM loss function and backpropagation

I'm trying to understand the connection between loss function and backpropagation. From what I understood until now, backpropagation is used to get and update matrices and bias used in forward ...
3
votes
4answers
126 views

Backpropagation: Relevance of the error signal of a neuron

During my quest to understand back propagation in a more rigorous approach I have come across with the definition of error signal of a neuron which is defined as follows for the $j^{\text{th}}$ neuron ...
1
vote
1answer
219 views

Backpropagation with log likelihood cost function and softmax activation

In the online book on neural networks by Michael Nielsen, in chapter 3, he introduces a new cost function called as log-likelihood function defined as below $$ C = -ln(a_y^L) $$ Suppose we have 10 ...
0
votes
0answers
32 views

Effect of ReLu derivative in convolution layer backpropagation

I'm trying to implement a CNN, as part of an academic project to learn how it works. The project is a SRCNN: a convolutional neural network that increase the resolution of images. Following this ...
1
vote
0answers
24 views

How to backpropogate error from convolutional layer with respect to the input when using multiple channels

I have been attempting to implement a Convolutional Neural Network in python and have run into a bit of a roadblock. When backpropogating the error in a convolutional layer let us say that we receive ...
1
vote
0answers
16 views

How does the Backpropagation through time work?

I have to write a paper on LSTMs and I want to explain why LSTMs exist in the first place. According to some papers and books because usual RNNs had problems with vanishing gradients and the LSTM has ...
0
votes
0answers
65 views

Forward and backward pass in Conv2D transpose Layer

I’ve several questions regarding the transposed convolution 2d layer. I’ve not been able to find a proper resource explaining the forward and backward pass. What I know (but not for sure) is, that ...
1
vote
2answers
45 views

BPTT vs Vanishing Gradient Problem

I know that BPTT is the method to apply Back Propagation on RNN. Which is works fine with RNN as it stops at certain point as changes approach to zero but isn't it the exact Vanishing Gradient ...
0
votes
0answers
89 views

Backpropagation Max-Pooling Layers: Multiple Maximum Values

I am currently implementing a CNN in plain numpy and have a brief question regarding a special case of the backpropagation for a max-pool layer: While it is clear that the gradient with respect to ...
1
vote
1answer
251 views

Backpropagation in a convolutional neural network with stride and padding

So i am trying to learn backpropagation of convolutional neural networks. A lot of articles only cover convolutions without a stride and a padding variable, so i decided to try it on my own. For ...
2
votes
0answers
26 views

Computing derivatives for backpropagation across a convolution step

This will be a long post, but I hope it'll be instructive to anyone else in my position. I'm trying to find how the derivatives of the loss function are calculated with respect to the kernels and ...
1
vote
1answer
27 views

Reason behind the sum of rate factors for calculating cost function derivative

Suppose we have a network of neurons like below: We make a little change in weight w[l][j][k] on our network, and it can make change on our cost function from ...
0
votes
0answers
33 views

What are the main reasons which does not cause the training error of yolov2 to not diminish?

I am using https://github.com/thtrieu/darkflow yolov2 for detecting and classifying the images of passport. There are 8 classes and all the objects are passport details like name, father name, mother ...

1
2 3 4 5