106 votes
Accepted

Backprop Through Max-Pooling Layers?

There is no gradient with respect to non maximum values, since changing them slightly does not affect the output. Further the max is locally linear with slope 1, with respect to the input that ...
user avatar
  • 1,178
23 votes
Accepted

Deep Neural Network - Backpropogation with ReLU

Working definitions of ReLU function and its derivative: $ReLU(x) = \begin{cases} 0, & \text{if } x < 0, \\ x, & \text{otherwise}. \end{cases}$ $\frac{d}{dx} ReLU(x) = \begin{cases} ...
user avatar
  • 27.4k
21 votes
Accepted

What do "compile", "fit", and "predict" do in Keras sequential models?

Let's first see what we need to do when we want to train a model. First, we want to decide a model architecture, this is the number of hidden layers and activation functions, etc. (compile) Secondly,...
user avatar
  • 8,518
18 votes

Gradients for bias terms in backpropagation

I would like to explain the meaning of db2=np.sum(dz2,axis=0,keepdims=True) as it also confused me once and it didn't get answered. The derivative of ...
user avatar
  • 532
17 votes
Accepted

back propagation in CNN

A convolution employs a weight sharing principle which will complicate the mathematics significantly but let's try to get through the weeds. I am drawing most of my explanation from this source. ...
user avatar
  • 8,518
15 votes

Guidelines for selecting an optimizer for training neural networks

AdaGrad penalizes the learning rate too harshly for parameters which are frequently updated and gives more learning rate to sparse parameters, parameters that are not updated as frequently. In several ...
user avatar
15 votes
Accepted

How does Gradient Descent and Backpropagation work together?

First, remember that the derivative of a function gives the direction in which the function increases, and its negative, the direction in which the function decreases. Training a model is just ...
user avatar
13 votes
Accepted

When are weights updated in CNN?

Whenever you train the network using batch means that you have chosen to train using batch gradient descent. There are three variants for gradient descent algorithm: Gradient Descent Stochastic ...
user avatar
  • 13.4k
13 votes
Accepted

Sliding window leads to overfitting in LSTM?

Although the previous answer by @Imran is correct, I feel it necessary to add a caveat: there are applications out there where people do feed a sliding window in to an LSTM. For example, here, for ...
user avatar
12 votes
Accepted

Gradients for bias terms in backpropagation

The bias term is very simple, which is why you often don't see it calculated. In fact db2 = dz2 So your update rules for bias on a single item are: ...
user avatar
  • 27.4k
12 votes
Accepted

Understanding how convolutional layers work

What are the filters? A filter/kernel is a set of learnable weights which are learned using the backpropagation algorithm. You can think of each filter as storing a single template/pattern. When you ...
user avatar
11 votes
Accepted

Question about bias in Convolutional Networks

Bias operates per virtual neuron, so there is no value in having multiple bias inputs where there is a single output - that would equivalent to just adding up the different bias weights into a single ...
user avatar
  • 27.4k
11 votes
Accepted

How to update bias and bias's weight using backpropagation algorithm

There should be a bias weight for each virtual neuron as it controls the threshold at which the neuron responds to combined input. So if your hidden layer has 100 neurons, that is 100 bias weights for ...
user avatar
  • 27.4k
11 votes

How does backpropagation works through Max Pooling layer when doing a batch?

When a neural network processes a batch, all activation values for each layer are calculated for each example (maybe in parallel per example if library and hardware support it). Those values are ...
user avatar
  • 27.4k
7 votes

Backprop Through Max-Pooling Layers?

Max Pooling So suppose you have a layer P which comes on top of a layer PR. Then the forward pass will be something like this: $ P_i = f(\sum_j W_{ij} PR_j)$, where $P_i$ is the activation of the ...
user avatar
7 votes

Creating neural net for xor function

A network with one hidden layer containing two neurons should be enough to seperate the XOR problem. The first neuron acts as an OR gate and the second one as a NOT AND gate. Add both the neurons and ...
user avatar
7 votes
Accepted

Creating neural net for xor function

Yes, there is a reason. It has to do with how you initialize your weights. There are 16 local minimums that have the highest probability of converging between 0.5 - 1. Here is a paper that ...
user avatar
  • 86
7 votes
Accepted

How to update weights in a neural network using gradient descent with mini-batches?

Let us say that the output of one neural network given it's parameters is $$f(x;w)$$ Let us define the loss function as the squared L2 loss (in this case). $$L(X,y;w) = \frac{1}{2n}\sum_{i=0}^{n}[f(...
user avatar
7 votes
Accepted

Correct number of biases in CNN

As you say, both approaches are used. It's called tied biases if you use one bias per convolutional filter/kernel ((3x5x5 + 1)x32 overall parameters in your example) and untied biases if you use one ...
user avatar
7 votes
Accepted

Backpropgating error to emedding matrix

An embedding layer is in fact a linear layer. It maps the input, using a matrix multiplication, to the output, without any activation function after the multiplication. Therefore, the backpropagation ...
user avatar
  • 5,674
7 votes
Accepted

How many times is backprop used in epoch?

It depends on the type of gradient descent or respectively your batch size: One epoch means that your neural net (NN) has applied the forward pass on all examples of your training data, i.e. it has "...
user avatar
  • 4,877
6 votes

In Neural Nets, why Use Gradient Methods as Opposed to Other Metaheuristics?

It would be a waste of information; the gradient is available, so use it and save time. There is reason to believe that the local optima are good; see, for example, Choromanska et al. (notes). Over-...
user avatar
  • 10.4k
6 votes

Guidelines for selecting an optimizer for training neural networks

My personal approach is to pick the optimizer that is newest (i.e. newest-published-in-a-peer-reviewed-journal), because they usually report results on standard datasets, or beat state of the art, or ...
user avatar
  • 231
6 votes
Accepted

Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training?

Yes the ReLU second order derivative is 0. Technically, neither $\frac{dy}{dx}$ nor $\frac{d^2y}{dx^2}$ are defined at $x=0$, but we ignore that - in practice an exact $x=0$ is rare and not especially ...
user avatar
  • 27.4k
6 votes
Accepted

What is the dimensionality of the bias term in neural networks?

As per the general case, the bias vector must have the same dimensions as the output vector. Please, have a look at this excellent presentation: In this example by M.Görner, there are 10 classes, so ...
user avatar
  • 179
6 votes

Sliding window leads to overfitting in LSTM?

LSTMs do not require a sliding window of inputs. They can remember what they have seen in the past, and if you feed in training examples one at a time they will choose the right size window of inputs ...
user avatar
  • 2,341
6 votes

Differences between gradient calculated by different reduction methods in PyTorch

Let's start by just recalling what each of these means. Reduction 'none' means compute batch_size gradient updates independently for the loss with respect to each ...
user avatar
  • 171
5 votes

Backprop Through Max-Pooling Layers?

@Shinvu's answer is well written, I would like to point to a video that explains the gradient of Max() operation and this within a computational graph which is quick to grasp.! while implementing the ...
user avatar
  • 318
5 votes
Accepted

Back-propagation through max pooling layers

If this is correct then every "neuron" of the pooling layer has the same gradient? No. It depends on the weights and activation function. And most typically the weights are different from the first ...
user avatar
5 votes
Accepted

How to apply the gradient of softmax in backprop

After further working on this, I figured out that: The homework implementation combines softmax with cross entropy loss as a matter of choice, while my choice of keeping softmax separate as an ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible