# Tag Info

91

There is no gradient with respect to non maximum values, since changing them slightly does not affect the output. Further the max is locally linear with slope 1, with respect to the input that actually achieves the max. Thus, the gradient from the next layer is passed back to only that neuron which achieved the max. All other neurons get zero gradient. So ...

21

Working definitions of ReLU function and its derivative: $ReLU(x) = \begin{cases} 0, & \text{if } x < 0, \\ x, & \text{otherwise}. \end{cases}$ $\frac{d}{dx} ReLU(x) = \begin{cases} 0, & \text{if } x < 0, \\ 1, & \text{otherwise}. \end{cases}$ The derivative is the unit step function. This does ignore a problem at $x=0$, ...

19

Let's first see what we need to do when we want to train a model. First, we want to decide a model architecture, this is the number of hidden layers and activation functions, etc. (compile) Secondly, we will want to train our model to get all the paramters to the correct value to map our inputs to our outputs. (fit) Lastly, we will want to use this model ...

15

I would like to explain the meaning of db2=np.sum(dz2,axis=0,keepdims=True) as it also confused me once and it didn't get answered. The derivative of L (loss) w.r.t. b is the upstream derivative multiplied with the local derivate: $$\frac{ \partial L}{\partial \mathbf{b}} = \frac{ \partial L}{\partial Z} \frac{ \partial Z}{\partial \mathbf{b}}$$ If we ...

15

A convolution employs a weight sharing principle which will complicate the mathematics significantly but let's try to get through the weeds. I am drawing most of my explanation from this source. Forward pass As you observed the forward pass of the convolutional layer can be expressed as $x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$ ...

12

Although the previous answer by @Imran is correct, I feel it necessary to add a caveat: there are applications out there where people do feed a sliding window in to an LSTM. For example, here, for framing forecasting as a supervised learning problem. If your data are not very rich, then you may find that any LSTM at all overfits. There are a lot of ...

12

What are the filters? A filter/kernel is a set of learnable weights which are learned using the backpropagation algorithm. You can think of each filter as storing a single template/pattern. When you convolve this filter across the corresponding input, you are basically trying to find out the similarity between the stored template and different locations in ...

11

There should be a bias weight for each virtual neuron as it controls the threshold at which the neuron responds to combined input. So if your hidden layer has 100 neurons, that is 100 bias weights for that layer. Same applies to each layer. There are usually two different approaches taken when implementing bias. You can do one or the other: As a separate ...

11

Whenever you train the network using batch means that you have chosen to train using batch gradient descent. There are three variants for gradient descent algorithm: Gradient Descent Stochastic Gradient Descent Batch Gradient Descent The first one passes the whole data through the network and finds the error rate for all of them and finds the gradients ...

10

When a neural network processes a batch, all activation values for each layer are calculated for each example (maybe in parallel per example if library and hardware support it). Those values are stored for possible later use - i.e. one value per activation per example in the batch, they are not aggregated in any way During back propagation, those activation ...

10

The bias term is very simple, which is why you often don't see it calculated. In fact db2 = dz2 So your update rules for bias on a single item are: b2 += -alpha * dz2 and b1 += -alpha * dz1 In terms of the maths, if your loss is $J$, and you know $\frac{\partial J}{\partial z_i}$ for a given neuron $i$ which has bias term $b_i$ . . . $$\frac{\partial ... 9 AdaGrad penalizes the learning rate too harshly for parameters which are frequently updated and gives more learning rate to sparse parameters, parameters that are not updated as frequently. In several problems often the most critical information is present in the data that is not as frequent but sparse. So if the problem you are working on deals with sparse ... 8 Bias operates per virtual neuron, so there is no value in having multiple bias inputs where there is a single output - that would equivalent to just adding up the different bias weights into a single bias. In the feature maps that are the output of the first hidden layer, the colours are no longer kept separate*. Effectively each feature map is a "channel" ... 7 Let us say that the output of one neural network given it's parameters is$$f(x;w)$$Let us define the loss function as the squared L2 loss (in this case).$$L(X,y;w) = \frac{1}{2n}\sum_{i=0}^{n}[f(X_i;w)-y_i]^2 In this case the batchsize will be denoted as $n$. Essentially what this means is that we iterate over a finite subset of samples with the size of ...

7

Yes, there is a reason. It has to do with how you initialize your weights. There are 16 local minimums that have the highest probability of converging between 0.5 - 1. Here is a paper that analyses the xor problem.

7

First, remember that the derivative of a function gives the direction in which the function increases, and its negative, the direction in which the function decreases. Training a model is just minimising the loss function, and to minimise you want to move in the negative direction of the derivative. Back-propagation is the process of calculating the ...

6

Max Pooling So suppose you have a layer P which comes on top of a layer PR. Then the forward pass will be something like this: $P_i = f(\sum_j W_{ij} PR_j)$, where $P_i$ is the activation of the ith neuron of the layer P, f is the activation function and W are the weights. So if you derive that, by the chain rule you get that the gradients flow as ...

6

A network with one hidden layer containing two neurons should be enough to seperate the XOR problem. The first neuron acts as an OR gate and the second one as a NOT AND gate. Add both the neurons and if they pass the treshold it's positive. You can just use linear decision neurons for this with adjusting the biases for the tresholds. The inputs of the NOT ...

6

It would be a waste of information; the gradient is available, so use it and save time. There is reason to believe that the local optima are good; see, for example, Choromanska et al. (notes). Over-optimizing for the training set leads to worse generalization, so sometimes we deliberately don't even try by stopping early. Probably the best free lunch in ...

6

As per the general case, the bias vector must have the same dimensions as the output vector. Please, have a look at this excellent presentation: In this example by M.Görner, there are 10 classes, so is bias dimension. Once inputs are multiplied by weights, the bias is added pointwise (it is 'broadcasted'). And that's pretty much it.

6

LSTMs do not require a sliding window of inputs. They can remember what they have seen in the past, and if you feed in training examples one at a time they will choose the right size window of inputs to remember on their own. LSTM's are already prone to overfitting, and if you feed in lots of redundant data with a sliding window then yes, they are likely to ...

5

If this is correct then every "neuron" of the pooling layer has the same gradient? No. It depends on the weights and activation function. And most typically the weights are different from the first neuron of the pooling layer to the FC layer as from the second layer of the pooling layer to the FC layer. So typically you will have a situation like: $FC_i =... 5 Yes the ReLU second order derivative is 0. Technically, neither$\frac{dy}{dx}$nor$\frac{d^2y}{dx^2}$are defined at$x=0$, but we ignore that - in practice an exact$x=0$is rare and not especially meaningful, so this is not a problem. Newton's method does not work on the ReLU transfer function because it has no stationary points. It also doesn't work ... 5 As you say, both approaches are used. It's called tied biases if you use one bias per convolutional filter/kernel ((3x5x5 + 1)x32 overall parameters in your example) and untied biases if you use one bias per kernel and output location ((3x5x5 + OxO)x32 overall parameters in your example). Untied biases increase the capacity of your model, so they can be ... 5 After further working on this, I figured out that: The homework implementation combines softmax with cross entropy loss as a matter of choice, while my choice of keeping softmax separate as an activation function is also valid. The homework implementation is indeed missing the derivative of softmax for the backprop pass. The gradient of softmax with respect ... 5 To answer the first question about why we need the learning rate even if we have momentum, let's consider an example in which we are not using the momentum term. The weight update is therefore:$ \Delta w_{ij} = \frac{\partial E}{\partial w_{ij}} \cdot l $where:$ \Delta w_{ij} $is the weight update$ \frac{\partial E}{\partial w_{ij}} $is the ... 5 It depends on the type of gradient descent or respectively your batch size: One epoch means that your neural net (NN) has applied the forward pass on all examples of your training data, i.e. it has "seen" all training data. Now to do so you have at least two options (let$n\$ be the number of samples in your training data): You can either run backprop after ...

4

My personal approach is to pick the optimizer that is newest (i.e. newest-published-in-a-peer-reviewed-journal), because they usually report results on standard datasets, or beat state of the art, or both. When I use Caffe for example, I always use Adam.

4

If you are using basic gradient descent (with no other optimisation, such as momentum), and a minimal network 2 inputs, 2 hidden neurons, 1 output neuron, then it is definitely possible to train it to learn XOR, but it can be quite tricky and unreliable. You may need to adjust learning rate. Most usual mistake is to set it too high, so the network will ...

4

in what sense applying the transpose is like "moving the error backwards"? It isn't. Or at least you shouldn't think too hard about the analogy. What you are actually doing is calculating the gradient - or partial derivative - of the error/cost term with respect to the activations and weights in the network. This is done by repeatedly applying the chain ...

Only top voted, non community-wiki answers of a minimum length are eligible