98

There is no gradient with respect to non maximum values, since changing them slightly does not affect the output. Further the max is locally linear with slope 1, with respect to the input that actually achieves the max. Thus, the gradient from the next layer is passed back to only that neuron which achieved the max. All other neurons get zero gradient. So in ...


21

Working definitions of ReLU function and its derivative: $ReLU(x) = \begin{cases} 0, & \text{if } x < 0, \\ x, & \text{otherwise}. \end{cases}$ $\frac{d}{dx} ReLU(x) = \begin{cases} 0, & \text{if } x < 0, \\ 1, & \text{otherwise}. \end{cases}$ The derivative is the unit step function. This does ignore a problem at $x=0$, ...


20

Let's first see what we need to do when we want to train a model. First, we want to decide a model architecture, this is the number of hidden layers and activation functions, etc. (compile) Secondly, we will want to train our model to get all the paramters to the correct value to map our inputs to our outputs. (fit) Lastly, we will want to use this model ...


17

I would like to explain the meaning of db2=np.sum(dz2,axis=0,keepdims=True) as it also confused me once and it didn't get answered. The derivative of L (loss) w.r.t. b is the upstream derivative multiplied with the local derivate: $$ \frac{ \partial L}{\partial \mathbf{b}} = \frac{ \partial L}{\partial Z} \frac{ \partial Z}{\partial \mathbf{b}} $$ If we ...


15

A convolution employs a weight sharing principle which will complicate the mathematics significantly but let's try to get through the weeds. I am drawing most of my explanation from this source. Forward pass As you observed the forward pass of the convolutional layer can be expressed as $x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$ ...


13

Although the previous answer by @Imran is correct, I feel it necessary to add a caveat: there are applications out there where people do feed a sliding window in to an LSTM. For example, here, for framing forecasting as a supervised learning problem. If your data are not very rich, then you may find that any LSTM at all overfits. There are a lot of ...


12

Whenever you train the network using batch means that you have chosen to train using batch gradient descent. There are three variants for gradient descent algorithm: Gradient Descent Stochastic Gradient Descent Batch Gradient Descent The first one passes the whole data through the network and finds the error rate for all of them and finds the gradients ...


12

What are the filters? A filter/kernel is a set of learnable weights which are learned using the backpropagation algorithm. You can think of each filter as storing a single template/pattern. When you convolve this filter across the corresponding input, you are basically trying to find out the similarity between the stored template and different locations in ...


11

AdaGrad penalizes the learning rate too harshly for parameters which are frequently updated and gives more learning rate to sparse parameters, parameters that are not updated as frequently. In several problems often the most critical information is present in the data that is not as frequent but sparse. So if the problem you are working on deals with sparse ...


11

There should be a bias weight for each virtual neuron as it controls the threshold at which the neuron responds to combined input. So if your hidden layer has 100 neurons, that is 100 bias weights for that layer. Same applies to each layer. There are usually two different approaches taken when implementing bias. You can do one or the other: As a separate ...


11

The bias term is very simple, which is why you often don't see it calculated. In fact db2 = dz2 So your update rules for bias on a single item are: b2 += -alpha * dz2 and b1 += -alpha * dz1 In terms of the maths, if your loss is $J$, and you know $\frac{\partial J}{\partial z_i}$ for a given neuron $i$ which has bias term $b_i$ . . . $$\frac{\partial ...


11

First, remember that the derivative of a function gives the direction in which the function increases, and its negative, the direction in which the function decreases. Training a model is just minimising the loss function, and to minimise you want to move in the negative direction of the derivative. Back-propagation is the process of calculating the ...


10

Bias operates per virtual neuron, so there is no value in having multiple bias inputs where there is a single output - that would equivalent to just adding up the different bias weights into a single bias. In the feature maps that are the output of the first hidden layer, the colours are no longer kept separate*. Effectively each feature map is a "channel" ...


10

When a neural network processes a batch, all activation values for each layer are calculated for each example (maybe in parallel per example if library and hardware support it). Those values are stored for possible later use - i.e. one value per activation per example in the batch, they are not aggregated in any way During back propagation, those activation ...


7

Let us say that the output of one neural network given it's parameters is $$f(x;w)$$ Let us define the loss function as the squared L2 loss (in this case). $$L(X,y;w) = \frac{1}{2n}\sum_{i=0}^{n}[f(X_i;w)-y_i]^2$$ In this case the batchsize will be denoted as $n$. Essentially what this means is that we iterate over a finite subset of samples with the size of ...


7

Yes, there is a reason. It has to do with how you initialize your weights. There are 16 local minimums that have the highest probability of converging between 0.5 - 1. Here is a paper that analyses the xor problem.


6

Max Pooling So suppose you have a layer P which comes on top of a layer PR. Then the forward pass will be something like this: $ P_i = f(\sum_j W_{ij} PR_j)$, where $P_i$ is the activation of the ith neuron of the layer P, f is the activation function and W are the weights. So if you derive that, by the chain rule you get that the gradients flow as ...


6

A network with one hidden layer containing two neurons should be enough to seperate the XOR problem. The first neuron acts as an OR gate and the second one as a NOT AND gate. Add both the neurons and if they pass the treshold it's positive. You can just use linear decision neurons for this with adjusting the biases for the tresholds. The inputs of the NOT ...


6

It would be a waste of information; the gradient is available, so use it and save time. There is reason to believe that the local optima are good; see, for example, Choromanska et al. (notes). Over-optimizing for the training set leads to worse generalization, so sometimes we deliberately don't even try by stopping early. Probably the best free lunch in ...


6

Yes the ReLU second order derivative is 0. Technically, neither $\frac{dy}{dx}$ nor $\frac{d^2y}{dx^2}$ are defined at $x=0$, but we ignore that - in practice an exact $x=0$ is rare and not especially meaningful, so this is not a problem. Newton's method does not work on the ReLU transfer function because it has no stationary points. It also doesn't work ...


6

As per the general case, the bias vector must have the same dimensions as the output vector. Please, have a look at this excellent presentation: In this example by M.Görner, there are 10 classes, so is bias dimension. Once inputs are multiplied by weights, the bias is added pointwise (it is 'broadcasted'). And that's pretty much it.


6

LSTMs do not require a sliding window of inputs. They can remember what they have seen in the past, and if you feed in training examples one at a time they will choose the right size window of inputs to remember on their own. LSTM's are already prone to overfitting, and if you feed in lots of redundant data with a sliding window then yes, they are likely to ...


6

It depends on the type of gradient descent or respectively your batch size: One epoch means that your neural net (NN) has applied the forward pass on all examples of your training data, i.e. it has "seen" all training data. Now to do so you have at least two options (let $n$ be the number of samples in your training data): You can either run backprop after ...


5

If this is correct then every "neuron" of the pooling layer has the same gradient? No. It depends on the weights and activation function. And most typically the weights are different from the first neuron of the pooling layer to the FC layer as from the second layer of the pooling layer to the FC layer. So typically you will have a situation like: $ FC_i =...


5

My personal approach is to pick the optimizer that is newest (i.e. newest-published-in-a-peer-reviewed-journal), because they usually report results on standard datasets, or beat state of the art, or both. When I use Caffe for example, I always use Adam.


5

As you say, both approaches are used. It's called tied biases if you use one bias per convolutional filter/kernel ((3x5x5 + 1)x32 overall parameters in your example) and untied biases if you use one bias per kernel and output location ((3x5x5 + OxO)x32 overall parameters in your example). Untied biases increase the capacity of your model, so they can be ...


5

After further working on this, I figured out that: The homework implementation combines softmax with cross entropy loss as a matter of choice, while my choice of keeping softmax separate as an activation function is also valid. The homework implementation is indeed missing the derivative of softmax for the backprop pass. The gradient of softmax with respect ...


5

To answer the first question about why we need the learning rate even if we have momentum, let's consider an example in which we are not using the momentum term. The weight update is therefore: $ \Delta w_{ij} = \frac{\partial E}{\partial w_{ij}} \cdot l $ where: $ \Delta w_{ij} $ is the weight update $ \frac{\partial E}{\partial w_{ij}} $ is the ...


5

Let's start by just recalling what each of these means. Reduction 'none' means compute batch_size gradient updates independently for the loss with respect to each input in the batch and then apply (the composition of) them. Reduction 'mean' and 'sum' mean apply the respective operations and the take the gradient with respect to this one value. Now, let's ...


5

Note that a CNN is a feed-forward neural network. Thus, if you understand how to perform backpropagation in feed-forward neural networks, you have it for CNNs. A convolution layer can be understood as a fully connected layer, with the constraints that several edge weights are identical and many edge weights are set to 0. You can also build a pooling layer in ...


Only top voted, non community-wiki answers of a minimum length are eligible