Questions tagged [backpropagation]

Use for questions about Backpropagation, which is commonly used in training Neural Networks in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
1
vote
0answers
26 views

Why can't a multi-layer linear neural network fit this linear function data?

I am learning to implement a neural network with gradient descent, and encountered this problem, please. Using the target function ...
2
votes
0answers
14 views

Derive backpropagation for PreLU

I want to derive the back propagation functions for the Parametric Relu activation function which is defined as follows: $$ h_a(x) = \text{max}(ax, x) $$ I want to derive $ \frac{\partial L}{\partial ...
3
votes
0answers
15 views

Homework/class help: Backward propagation of max pooling if each element in an array determines more than one value?

(This isn't actually my homework, and in fact wasn't addressed in my homework, but I was confused about this because my homework hadn't addressed this) For example if I have an array: And I do max ...
0
votes
0answers
8 views

Back propagation - updating basis values

When doing backpropergation using gradient decent (batch descent), the dW / DL ends up with a matrix the same size a W regardless of the size of the batch. The dB / dL (basis adjustment values) ends ...
4
votes
1answer
132 views

What changes is the Neural Network back-propagation algorithm doing on the weights?

I have seen the formula for back-propagation algorithm for neural network error minimization, but I am not quite sure about what changes it is performing on the weights individually. Let us suppose a ...
0
votes
1answer
17 views

How does TensorFlow handle multiple samples?

Say the mini-batch has $N$ samples $(x, y)$, how will tensorflow utilize this $N$ samples to train the network. Will it do $N$ forward loop for each sample independently? Will it do $N$ backward ...
2
votes
1answer
44 views

Transferring the hidden state of a RNN to another RNN

I am using Reinforcement Learning to teach an AI an Austrian Card Game with imperfect information called Schnapsen. For different states of the game, I have different neural networks (which use ...
0
votes
1answer
25 views

Understanding intution behind sigmoid curve in the context of back propagation

I was trying to understand significance of S-shape of sigmoid / logistic function. The slope/derivative of sigmoid approaches zero for very large and very small input values. That is $σ'(z) ≈ 0$ for $...
0
votes
0answers
15 views

Differentiation of Learning Capabilities of Different Networks

I have a conceptual problem regarding the overall learning capability of a neural network differentiated by the different types of input that we can give to the network. Suppose that we have a ...
0
votes
0answers
12 views

Update function for NN with logistic and sofmax

Can anyone help me confirm my work or find resources on how to come up with the update function for each layer in the Neural Network for multi-class classification problem i.e I am using logistic as ...
0
votes
0answers
7 views

Novice machine learner wondering how to interpret big variance in batch error across batches in MNIST perceptron

I'm trying to get a better understanding of basic neural networks by implementing a little framework in C++. I've started with the classical MNIST exercise. I get to 91% accuracy on the test sample ...
1
vote
0answers
22 views

Backpropagation derivation for a convolutional layer

I am trying to derive the backpropagation for a single convolutional layer (padding layer is being implemented separately, so no padding argument for the convolutional layer). This layer is given $\...
1
vote
0answers
30 views

Batch normalization backpropagation doubts

I have recently studied the batch normalization layer and its backpropagation process, using as my main sources the original paper and this website showing part of the derivation process, but there is ...
4
votes
3answers
263 views

What backpropagation actually is?

I have a conceptual question due to terminology that bothers me. Is backpropagation algorithm a neural network training algorithm or is it just a recursive algorithm in order to calculate a Jacobian ...
1
vote
0answers
12 views

Jacobian calculation: Partial derivatives of network outputs respect to inner layers in feedforward neural network

I'm having hard times in deriving a Jacobian (derivative of final network outputs respect to all network parameters!) for a neural network that you see in the picture below. It's about two level ...
0
votes
0answers
8 views

Backpropagation in RNN in discrete visible units

Refer to https://www.reddit.com/r/MachineLearning/comments/40ldq6/generative_adversarial_networks_for_text/ Goodfellow said that we still don't have a way to use GANs in NLP because of its discrete (...
0
votes
0answers
10 views

Regarding the backpropagation

I was reading a GNN intro book which discussed backpropation in NNs shortly. I was particularly concerned about the following equation: This is different from what I know, that when the parameter ...
0
votes
0answers
20 views

Andrew Ng Deep Learning Gradient Descent of Softmax is just y_hat - y?

At about 8:30 in the video here: https://www.youtube.com/watch?v=ueO_Ph0Pyqk so for the given example with 4 classes and first ground truth y being [0,1,0,0] and y_hat being [0.3,0.2,0.1,0.4] for ...
0
votes
0answers
176 views

PyTorch backwards() call on loss function

Can someone confirm that a call to loss.backward() given loss defined with nn.MSELoss() if called in a loop like this: ...
1
vote
1answer
46 views

Force neural network to only product positive values

I have a custom neural network that has been written from scratch in python and also a dataset where negative target/response values are impossible, however my model sometimes produces negatives ...
1
vote
0answers
25 views

how to calculate parameters of an RNN using backpropagation

I'm trying to find out the two binary inputs are identical or not using RNN. my architecture is like this: I have the following functions: Where vT is the transpose of vector v and the activation ...
2
votes
1answer
39 views

How to train a deep neural network to return the input as it is?

The task is to train a neural network to return the input as it is, like X -> X or Y -> Y. The network should contain at ...
2
votes
1answer
58 views

Implementing computational graph and autograd for tensor and matrix

I am trying to implement a very simple deep learning framework like PyTorch in order to get a better understanding of computational graphs and automatic differentiation. I implemented an automatic ...
2
votes
2answers
39 views

Backpropagation During Neural Networks Training - Units while updating weights

I found this article that describes how neural networks work. This paragraph near the end caught my eye and explains how weights are updated: So we see that $\theta_i := \theta_i + \nabla\theta_i$ ...
1
vote
0answers
57 views

How does Keras optimization for a network with multiple outputs

I currently have a neural network that takes in 3 numbers as inputs and outputs 3 numbers. I've attached a picture of the network below and my code is accessible through the following link: [Google ...
2
votes
1answer
730 views

How batch normalization layer resolve the vanishing gradient problem?

According to this article: https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484 The vanishing gradient problem occurs when using the sigmoid ...
0
votes
0answers
9 views

Backpropagation for Network Architecture Search (NAS)

in normal backpropagation, there is only a single edge in between two nodes. However in GDAS, there are multiple parallel edges in between two nodes. So, how to perform backpropagation for Network ...
0
votes
0answers
40 views

Back propagation process of RNN?

I'm learning how to use the Recurrent Neural Network model (RNN). I'm not entirely sure about the feed-forward procedure in RNN. It includes, for example, input, hidden state, and output. As far as I ...
1
vote
0answers
20 views

Back propagation to find the value of z

I hope all of you are doing well. I am a high school student, studying machine learning for my interest.I am studying the Backpropagation Algorithm in recent days. I got stuck in a problem: After the ...
0
votes
0answers
21 views

Getting gradient for gradCam in pytorch

I am using forward and backward hook in my pytorch densenet121 model. I set requires_grad to False at the time of training. ...
0
votes
0answers
9 views

Matrix dimensions issue on calculating Softmax Derivatives

I am trying to get the matrix dimensions right for computing derivative of a two layers network where the last layer is softmax function. For simplicity I am only interested to get derivatives of W2 w....
0
votes
0answers
99 views

Backprop with relu and softmax in matlab

I am trying to implement a network for a classification task and I am kinda struggling with backpropagation. The network should classify MNIST dataset (0 - 9). As a training set, I have 4500 images of ...
3
votes
1answer
38 views

Recurrent Neural Network (RNN) Vanishing gradient problem - Why does it affect earlier timesteps more?

I understand the concept of backpropagation in standard neural networks and backpropagation through time with RNNs, why this causes exponentially smaller gradients at earlier time steps and most of ...
0
votes
1answer
35 views

Differentiating vector with different operation on each elements

I have some idea about how backpropagation would work for a loss function like: loss=summation(predicted-true)^2 Where predicted and true are vectors of the same ...
0
votes
1answer
22 views

Is the usage of the "momentum" significiantly superior to the conventional weight update

The "momentum" adds a little of the history of the last weight updates to the actual update, with diminishing weight history (older momentum shares get smaller). Is it significiantly ...
0
votes
0answers
34 views

GRU - why gamma is close to 0 or 1 resolves the vanishing gradient problem?

Background It is explained that in GRU (Gated Recurrent Unit), the gate value $\Gamma$ close to zero mitigates the vanishing gradient problem. Coursera Sequence Model by Andrew Ng - Week 1 Gated ...
1
vote
0answers
36 views

Dummy Variables of Weights in RNN Backpropagation Through Time

In the deep learning book RNN chapter (https://www.deeplearningbook.org/contents/rnn.html), it is mentioned that - To resolve this ambiguity, we introduce dummy variables $W^{(t)}$ that are defined to ...
0
votes
0answers
26 views

Gradient calculation analysis

I'm using VGG16 pretrained architecture for classification and visualization of result using Guided Backpropagation technique. I have used tensorflow code for calaulating the input gradientSharing ...
1
vote
1answer
28 views

What issue is there, when training this network with gradient descent? [closed]

Suppose we have the following fully connected network made of perceptrons with a sign function as the activation unit, what issue arises, when trying to train this network with gradient descent?
0
votes
1answer
27 views

Vanishing gradient problem

In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers?
0
votes
1answer
32 views

Why is backpropagation used for finding the loss gradient?

I am relatively new to the world of machine learning. After getting a general idea of the concept, I tried creating a program for training a deep learning network from scratch. My goal was to use as ...
0
votes
0answers
172 views

Need help to understand backpropagation for gated recurrent units (GRU)

I'm stuck regarding the implementation of backpropagation in a binary classification task using a GRU. I wanted to know if someone could tell me how to proceed. I was able to understand how BP works ...
1
vote
0answers
29 views

Padding in Convolution Formula

Why is it that the formula for each element in a convolution between an image $I$ and a $k \times k$ sized kernel $K$ is $$ (I*K)_{ij}=\sum_{m=0}^{k-1}\sum_{n=0}^{k-1}I_{(i-m),(j-n)}K_{mn}=\sum_{m=0}^{...
0
votes
0answers
17 views

Understanding the convolution formula

According to several sources this formula, or the center originated version of it, is used to calculate an element of a convolution between an image $I$ and a kernel $K$ of size $k \times k$: $$ (I*K)...
0
votes
0answers
20 views

Decoupling the performance impact of batch sizes from their impact on training speed

We are training a sequence-to-sequence-model to do something like Named Entity Recognition using PyTorch. It turns out, we get the best results with batch size 1. However, this slows down training ...
1
vote
0answers
70 views

Tensor Backpropagation

I tried to make simple neural network layers as the following, including forward and backward propagation. Here is my reference. Firstly I assume an one layer FC: $Y = X \cdot W + B$ X is input, which ...
1
vote
1answer
43 views

Gradient passthough in PyTorch

I need to quantize the inputs, but the method (bucketize) I need to do so is indifferentiable. I can of course detach the tensor, but then I lose the flow of gradients to earlier weights. I guess ...
0
votes
0answers
309 views

Why divide by batch size when back-propagate from softmax + log loss

Question In neural network mini batch training, at the back-propagation from the (Softmax + cross entropy log loss) layer, the gradient is divided by the batch size. Please explain why need to do so. ...
1
vote
1answer
243 views

Problem with convergence of ReLu in MLP

I created neural network from scratch in python using only numpy and I'm playing with different activation functions. What I observed is quite weird and I would love to understand why this happens. ...
2
votes
0answers
127 views

How to implement a cascaded neural network in Keras where 1st NN's forward output is cloned twice to perform forward on the 2nd NN?

Using Keras, I am trying to reproduce a few basic results from a published paper. In this task, there are two neural networks - A & B, that are connected in a cascade formation, i.e. the output of ...

1
2 3 4 5 6