All Questions
Tagged with neural-network backpropagation
168 questions
1
vote
1
answer
135
views
Why don't we transpose $\delta^{l+1}$ in back propagation?
Using this neural network as an example:
The weight matrices are then
$$ W_0=[2\times4], W_1=[4\times4], W_2=[4\times2]$$
To find the error for the last layer, we use
$$ \delta^{[2]} = \nabla C \odot ...
- 113
1
vote
0
answers
29
views
Wich activation function for DQL
After many research, I still can't find a neat answer about this question:
When I found the loss of my state-action pair. I'm only backpropagating that loss true the network and setting all other ...
0
votes
0
answers
96
views
clarification on back-propagation calculations for a fully connected neural network
I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated.
Let's say my fully connected neural network looks like ...
1
vote
1
answer
162
views
How are the gradients of a Neural Network calculated just by matrix multiplication?
I would have expected some kind of derivative solving equation to be at work in order to back propagate the loss to each neuron. I hope my question is not too confused to answer.
In the network below, ...
- 207
0
votes
1
answer
47
views
Neural Network not learning when more than 1 training data is given
I am very new to neural networks and data science in general and wanted to try getting my hand in making a simple neural network in python.
I tried to make a neural network from scratch hoping to ...
0
votes
1
answer
156
views
Neural Net Backprop Weight updating Pseudo code help please
Here is my code for Backpropagation weight updating. It's a simple network with 1 hidden layer and 1 output neuron. The activation function of both hidden and output layer uses tanh. I propagate the ...
- 1
-3
votes
2
answers
524
views
Chain function in backpropagation
I'm reading a Neural Networks Tutorial. In order to answer my question you might have to take a brief look at it.
I understand everything until something they declare as "chain function":
...
- 23
0
votes
1
answer
146
views
how does gradient descent update weigths in neural network
Im currently trying to learn about back propagation, and it's going forward, but theres one thing that keeps me scratching my head, and doesnt really seems to be answered in any of the videos or ...
- 3
1
vote
0
answers
47
views
Backpropagation simple question [closed]
I have a few doubts in backpropagation....
1.) My first doubt is that we can write the cost function as all the neural elements involved from all layers and then we can differentiate with respect to ...
- 109
1
vote
0
answers
273
views
Generalized softmax derivative for implementation with any loss function
I am currently taking some deep learning and neural network (NN) courses, and in addition to performing the course work, am implementing my own "toolkit" of NN techniques to better my understanding of ...
- 11
3
votes
2
answers
61
views
Backpropagation with a different sized training set?
I'm trying to create a NN whose input is a (length m) array of 3d vectors $$\vec{x}_i = [x_{i,1},x_{i,2},x_{i,3}], \hspace{5mm}i=1:m $$
and whose output is a similarly sized array:
$$\vec{h}_{\theta,...
- 31
1
vote
0
answers
60
views
What's wrong with my backpropagation through time (BTT) calculation or how to multiple a scaled vector and a matrix without matching dimensions?
I am trying to make a pretty simple RNN from scracth, using only Numpy library of Python.
At this moment I am having troubles with BTT as I do not know how to proceed with situation when a ...
- 29
0
votes
1
answer
58
views
Gradient for starting Backpropagation
I was reading this nice tutorial about Pytorch's basics:
https://pytorch.org/tutorials/beginner/pytorch_with_examples.html
In the first example (pure Numpy), the author starts the backward phase by ...
- 103
4
votes
1
answer
3k
views
If mean absolute loss is not differentiable, how it can be used in neural networks? which majorly are trained using back-propagation
If Mean Absolute Error (MAE) loss is not differentiable, how can it be used in neural networks? which majorly are trained using back-propagation
I am wondering if MAE is not differentiable how they ...
- 143
5
votes
1
answer
3k
views
How many times is backprop used in epoch?
As I understand for the algorithms that use gradient descent we have to pass data to the algorithms multiple times so that the optimum is found.
So one epoch means that the forward-backprop (and ...
- 153
1
vote
1
answer
146
views
What is the layer above/below in a NN?
In the lecture notes of CS231n, it says (emphasis mine)
... There are three major sources of memory to keep track of:
From the intermediate volume sizes: These are the raw number of ...
- 113
3
votes
1
answer
1k
views
Interpreting Gradients and Partial Derivatives when training Neural Networks
I am trying to understand of purpose of partial differentiation in NN training by knowing how to interpret gradients and their partial derivatives. Below is my way of interpreting them so I would like ...
- 746
1
vote
2
answers
44
views
Justification for values used in backpropagation
I'm learning the method for backpropagation in adjusting weights. A generalization of a formula used to determine the change made to a respective weight is
where is the rate the total error changes ...
- 245
4
votes
3
answers
12k
views
Forward pass vs backward pass vs backpropagation
As mentioned in the question, I have some issues understanding what are the differences between those terms.
From what I have understood:
Forward pass: compute the output of the network given the ...
2
votes
2
answers
235
views
Hello, anyone able to direct me to a "cheat sheet" of Neural Network equations with legends?
Can anyone here can direct me to a site that provides a cheat sheet of equations for Neural Networks with a legend for the notation?
It can be on any and all aspects of NN, be it forward or back ...
1
vote
0
answers
37
views
Question regarding: Vectorization Math of Backpropagation in a Neural Network
Formula:
These are the formula I use for backpropagation from Brilliant:
Question:
If we consider a Neural Network with the structure (3,2):
And we would start calculating the derivative (for 1 ...
- 133
2
votes
1
answer
80
views
Problem with chain rule in softmax layer when differentiated separately
I have some problems with backpropagation in softmax output layer. I know how it should work but if I try to apply the chain rule in the classical way, I get different results compared to when Softmax ...
- 153
1
vote
0
answers
41
views
How to do Back Propagation Updation for below code?
Below is the code by Siraj Raval for implementing Neural Networks from Scratch. I have some doubts regarding the Code:
Why during updation he did W2 = W2 + L1.T.dot(L2_Delta). I Mean shouldn't it be ...
- 31
1
vote
0
answers
128
views
Neural Network from scratch: cost increasing over epochs
I'm trying to design a neural network from scratch. After training my neural network, I make a plot of the cost vs epochs, which I would expect to decrease throughout the runtime of the NN training, ...
- 11
2
votes
1
answer
101
views
Doubt in Derivation of Backpropagation
I was going through the derivation of backpropagation algorithm provided in this document (adding just for reference). I have doubt at one specific point in this derivation. The derivation goes as ...
- 175
0
votes
2
answers
1k
views
Backpropagation and Stochastic Gradient Descent(SGD)
Is Backpropagation a learning method or an optimisation method?
How are Backpropagation and Stochastic Gradient Descent(SGD) related to each other?
- 168
2
votes
1
answer
126
views
Backpropagation chain rule example
My question is in regards to an MIT course example.
The instructor delves into the backpropagation of this simple NN.
I have two questions.
Why do we seem to disregard the weights of the second ...
- 419
1
vote
1
answer
736
views
How CNN applies backpropagation to update its weights and biases?
I understand that the 3 main layers for CNN are convolutional layer, ReLU layer and pooling layer.
However, I do not understand how CNN updates its weights and biases using backpropagation.
I ...
- 101
4
votes
4
answers
528
views
Backpropagation: Relevance of the error signal of a neuron
During my quest to understand back propagation in a more rigorous approach I have come across with the definition of error signal of a neuron which is defined as follows for the $j^{\text{th}}$ neuron ...
- 41
1
vote
1
answer
1k
views
Backpropagation with log likelihood cost function and softmax activation
In the online book on neural networks by Michael Nielsen, in chapter 3, he introduces a new cost function called as log-likelihood function defined as below
$$ C = -ln(a_y^L) $$
Suppose we have 10 ...
- 724
0
votes
0
answers
356
views
Forward and backward pass in Conv2D transpose Layer
I’ve several questions regarding the transposed convolution 2d layer. I’ve not been able to find a proper resource explaining the forward and backward pass. What I know (but not for sure) is, that ...
- 101
3
votes
0
answers
123
views
Computing derivatives for backpropagation across a convolution step
This will be a long post, but I hope it'll be instructive to anyone else in my position. I'm trying to find how the derivatives of the loss function are calculated with respect to the kernels and ...
- 299
1
vote
1
answer
82
views
Reason behind the sum of rate factors for calculating cost function derivative
Suppose we have a network of neurons like below:
We make a little change in weight w[l][j][k] on our network, and it can make change on our cost function from ...
- 171
1
vote
1
answer
80
views
Neural network is getting partially trained
So I am writing my own neural network library using back-propagation as my training algorithm. Everything seems fine the error is getting decreased more and more at each iteration however when I am ...
0
votes
1
answer
358
views
How do I convert a summation equation to a vector equation (backpropagation)?
$$a_j^l=\sigma(\sum_{k} {w_j}_k^l {a}_k^{l-1}+b_j^l)$$
$$a^l=\sigma( w^l {a}^{l-1}+b^l)$$
In a resource I have been reading, the above equations describe the activation of a neurone. They have the ...
- 451
0
votes
1
answer
260
views
Confused with the derivation of the gradient descent update rule
I have been going over some theory for gradient descent. The source I am looking at said that the change in cost can be described by the following equation:
$$∆C=∇C∙∆w$$
where $∇C$ is the gradient ...
- 451
2
votes
1
answer
398
views
Layer notation for feed forward neural networks
Apologies in advance, for I have a fairly rudimentary question on the notations for studying Feed-Forward Neural Networks. Here is a nice schematic taken from this blog-post.
Here $x_i = f_i(W_i \...
- 133
0
votes
1
answer
49
views
What does this expression from gradient descent mean?
I am looking over some neural network theory and came across this equation, coupled with this description (gradient descent ball-valley analogy):
''let's think about what happens when we move the ...
- 451
1
vote
0
answers
83
views
Having problem in back propagation part for dimension
I was trying to build a neural network with single hidden layer from scratch. In back propagation part some problems have raised.
For calculating gradient of loss function with respect to weight in ...
- 123
0
votes
1
answer
139
views
Back propagation on matrix of weights
I am trying to implement a Neural Network for binary classification using python and numpy only.
My network structure is as follows:
input features: 2 [1X2] matrix
Hidden layer1: 5 neurons [2X5] ...
- 21
1
vote
0
answers
171
views
CartPole v1 - Simple backprop with 1 hidden layer
I'm trying to solve the CartPole-v1 problem from OpenAI by using backprop on a one-layer neural network - while updating the model at every time step using State action values (Q(s,a)). I'm unable to ...
3
votes
1
answer
4k
views
Derivation of backpropagation for Softmax
So, after a couple dozen tries I finally implemented a standalone nice and flashy softmax layer for my neural network in numpy. All works well, but I have a question regarding the maths part because ...
- 153
1
vote
1
answer
154
views
Backpropagation for noobs
I am trying to understand neural networks and how they work, by programming my own one from scratch in nodejs. Currently, i managed to build a network, that has weights, layers and neurons. I also ...
- 27
24
votes
1
answer
26k
views
How does Gradient Descent and Backpropagation work together?
Please forgive me as I am new to this. I have attached a diagram trying to model my understanding of neural network and Back-propagation? From videos on Coursera and resources online I formed the ...
- 355
2
votes
1
answer
144
views
Understanding backprop for softmax
I'm looking on a given solution of the first assignment of cs231n course.
Down below a snippet from the loss function.
I don't really understand lines 140-143. Can ...
- 141
0
votes
1
answer
127
views
Information about LTSM RNN backpropagation algorithm
I am attempting to make a LTSM RNN in python from scratch and I have completed the code for forward pass but I am struggling to find a clear outline of the equations I need to calculate to get the ...
- 37
0
votes
1
answer
390
views
What is the difference between reconstruction vs backpropagation?
I was following a tutorial on understanding Restricted Boltzmann Machines (RBMs) and I noticed that they used both the terms reconstruction and ...
- 257
1
vote
1
answer
153
views
Do we need to add the sigmoid derivative term in the final layer's error value?
I have been studying professor Andrew Ng's Machine Learning course on Coursera. Currently, I am trying to prove the formulas for backpropagation, which is mentioned in Week 5 (in this document). ...
- 13
0
votes
2
answers
2k
views
Using multiple machine learning algorithms together [closed]
I'm kinda new to machine learning and wanted to know if we could use multiple machine learning algorithms, for example, SVM and backpropagation together to solve a particular problem.
- 101
0
votes
2
answers
2k
views
How can I perform backpropagation directly in matrix form?
I had made a neural network library a few months ago, and I wasn't too familiar with matrices. So, instead of performing matrix dot products (between weights and inputs, then adding a bias matrix), I ...
user42078