Skip to main content
111 votes
Accepted

Backprop Through Max-Pooling Layers?

There is no gradient with respect to non maximum values, since changing them slightly does not affect the output. Further the max is locally linear with slope 1, with respect to the input that ...
abora's user avatar
  • 1,228
25 votes
Accepted

Deep Neural Network - Backpropogation with ReLU

Working definitions of ReLU function and its derivative: $ReLU(x) = \begin{cases} 0, & \text{if } x < 0, \\ x, & \text{otherwise}. \end{cases}$ $\frac{d}{dx} ReLU(x) = \begin{cases} ...
Neil Slater's user avatar
23 votes
Accepted

What do "compile", "fit", and "predict" do in Keras sequential models?

Let's first see what we need to do when we want to train a model. First, we want to decide a model architecture, this is the number of hidden layers and activation functions, etc. (compile) Secondly,...
JahKnows's user avatar
  • 8,946
20 votes

Gradients for bias terms in backpropagation

I would like to explain the meaning of db2=np.sum(dz2,axis=0,keepdims=True) as it also confused me once and it didn't get answered. The derivative of ...
oezguensi's user avatar
  • 602
20 votes
Accepted

How does Gradient Descent and Backpropagation work together?

First, remember that the derivative of a function gives the direction in which the function increases, and its negative, the direction in which the function decreases. Training a model is just ...
Escachator's user avatar
18 votes

Guidelines for selecting an optimizer for training neural networks

AdaGrad penalizes the learning rate too harshly for parameters which are frequently updated and gives more learning rate to sparse parameters, parameters that are not updated as frequently. In several ...
Santanu_Pattanayak's user avatar
17 votes
Accepted

back propagation in CNN

A convolution employs a weight sharing principle which will complicate the mathematics significantly but let's try to get through the weeds. I am drawing most of my explanation from this source. ...
JahKnows's user avatar
  • 8,946
14 votes
Accepted

When are weights updated in CNN?

Whenever you train the network using batch means that you have chosen to train using batch gradient descent. There are three variants for gradient descent algorithm: Gradient Descent Stochastic ...
Green Falcon's user avatar
  • 14.1k
14 votes
Accepted

Sliding window leads to overfitting in LSTM?

Although the previous answer by @Imran is correct, I feel it necessary to add a caveat: there are applications out there where people do feed a sliding window in to an LSTM. For example, here, for ...
StatsSorceress's user avatar
12 votes
Accepted

Question about bias in Convolutional Networks

Bias operates per virtual neuron, so there is no value in having multiple bias inputs where there is a single output - that would equivalent to just adding up the different bias weights into a single ...
Neil Slater's user avatar
12 votes
Accepted

Gradients for bias terms in backpropagation

The bias term is very simple, which is why you often don't see it calculated. In fact db2 = dz2 So your update rules for bias on a single item are: ...
Neil Slater's user avatar
12 votes
Accepted

Understanding how convolutional layers work

What are the filters? A filter/kernel is a set of learnable weights which are learned using the backpropagation algorithm. You can think of each filter as storing a single template/pattern. When you ...
Akshay Sehgal's user avatar
11 votes
Accepted

How to update bias and bias's weight using backpropagation algorithm

There should be a bias weight for each virtual neuron as it controls the threshold at which the neuron responds to combined input. So if your hidden layer has 100 neurons, that is 100 bias weights for ...
Neil Slater's user avatar
11 votes

How does backpropagation works through Max Pooling layer when doing a batch?

When a neural network processes a batch, all activation values for each layer are calculated for each example (maybe in parallel per example if library and hardware support it). Those values are ...
Neil Slater's user avatar
10 votes

Creating neural net for xor function

A network with one hidden layer containing two neurons should be enough to separate the XOR problem. The first neuron acts as an OR gate and the second one as a NOT AND gate. Add both the neurons and ...
Jan van der Vegt's user avatar
9 votes
Accepted

Creating neural net for xor function

Yes, there is a reason. It has to do with how you initialize your weights. There are 16 local minimums that have the highest probability of converging between 0.5 - 1. Here is a paper that analyses ...
Emil's user avatar
  • 106
8 votes
Accepted

Backpropgating error to emedding matrix

An embedding layer is in fact a linear layer. It maps the input, using a matrix multiplication, to the output, without any activation function after the multiplication. Therefore, the backpropagation ...
David Masip's user avatar
  • 6,101
8 votes
Accepted

How many times is backprop used in epoch?

It depends on the type of gradient descent or respectively your batch size: One epoch means that your neural net (NN) has applied the forward pass on all examples of your training data, i.e. it has "...
Jonathan's user avatar
  • 5,430
7 votes

Backprop Through Max-Pooling Layers?

Max Pooling So suppose you have a layer P which comes on top of a layer PR. Then the forward pass will be something like this: $ P_i = f(\sum_j W_{ij} PR_j)$, where $P_i$ is the activation of the ...
patapouf_ai's user avatar
7 votes
Accepted

Correct number of biases in CNN

As you say, both approaches are used. It's called tied biases if you use one bias per convolutional filter/kernel ((3x5x5 + 1)x32 overall parameters in your example) and untied biases if you use one ...
robintibor's user avatar
7 votes
Accepted

What Is Saturating Gradient Problem

If you use sigmoid-like activation functions, like sigmoid and tanh, after some epochs of training, the linear part of each neuron will have values that are very big or very small. This means that the ...
Green Falcon's user avatar
  • 14.1k
6 votes

Backprop Through Max-Pooling Layers?

@Shinvu's answer is well written, I would like to point to a video that explains the gradient of Max() operation and this within a computational graph which is quick to grasp.! while implementing the ...
Anu's user avatar
  • 328
6 votes

Creating neural net for xor function

If you are using basic gradient descent (with no other optimisation, such as momentum), and a minimal network 2 inputs, 2 hidden neurons, 1 output neuron, then it is definitely possible to train it to ...
Neil Slater's user avatar
6 votes

In Neural Nets, why Use Gradient Methods as Opposed to Other Metaheuristics?

It would be a waste of information; the gradient is available, so use it and save time. There is reason to believe that the local optima are good; see, for example, Choromanska et al. (notes). Over-...
Emre's user avatar
  • 10.5k
6 votes
Accepted

Back-propagation through max pooling layers

If this is correct then every "neuron" of the pooling layer has the same gradient? No. It depends on the weights and activation function. And most typically the weights are different from the first ...
patapouf_ai's user avatar
6 votes
Accepted

Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training?

Yes the ReLU second order derivative is 0. Technically, neither $\frac{dy}{dx}$ nor $\frac{d^2y}{dx^2}$ are defined at $x=0$, but we ignore that - in practice an exact $x=0$ is rare and not especially ...
Neil Slater's user avatar
6 votes
Accepted

What is the dimensionality of the bias term in neural networks?

As per the general case, the bias vector must have the same dimensions as the output vector. Please, have a look at this excellent presentation: In this example by M.Görner, there are 10 classes, so ...
tagoma's user avatar
  • 179
6 votes

Sliding window leads to overfitting in LSTM?

LSTMs do not require a sliding window of inputs. They can remember what they have seen in the past, and if you feed in training examples one at a time they will choose the right size window of inputs ...
Imran's user avatar
  • 2,381
6 votes

Differences between gradient calculated by different reduction methods in PyTorch

Let's start by just recalling what each of these means. Reduction 'none' means compute batch_size gradient updates independently for the loss with respect to each ...
an1lam's user avatar
  • 171
6 votes

How batch normalization layer resolve the vanishing gradient problem?

Batch Normalization (BN) does not prevent the vanishing or exploding gradient problem in a sense that these are impossible. Rather it reduces the probability for these to occur. Accordingly, the ...
Jonathan's user avatar
  • 5,430

Only top scored, non community-wiki answers of a minimum length are eligible