10
$\begingroup$

Let's assume that we are using a batch size of 100 samples for learning.

So in every batch, the weight of every neuron (and bias, etc) is being updated by adding the minus of the learning rate * the average error value that we found using the 100 samples * the derivative of the error function with the respect to the current neuron weight that is being updated.

Now, when we use a Max Pool layer, how can we compute the derivative over this layer? In every sample that we feed forward, a different pixel (let's say) is chosen as the max, so when we backpropagate over 100 samples in which every time a different path was chosen, how can we do it? A solution I have in mind is to remember every pixel that was chosen as the maximum, and then maybe split the derivative over all the max-pixels. Is this what's being done?

Improve this question
$\endgroup$
  • $\begingroup$ I have same question. when BP with only one sample, it's clear that only the biggest element's derivative is not zero, but when BP with samples in batch, different sample may cause different position of biggest element, can we just calculate average derivative of each parameter as usual (just add ∂L/∂wi of every sample and divided by batch size)? $\endgroup$ – Shaotao Li Aug 31 '18 at 9:47
9
$\begingroup$

When a neural network processes a batch, all activation values for each layer are calculated for each example (maybe in parallel per example if library and hardware support it). Those values are stored for possible later use - i.e. one value per activation per example in the batch, they are not aggregated in any way

During back propagation, those activation values are used as one of the numerical sources to calculate gradients, along with gradients calculated so far working backwards and the connecting weights. Like forward propagation, back propagation is applied per example, it does not work with averaged or summed values. Only when all examples have been processed do you work with the summed or averaged gradients for the batch.

This applies equally to max pool layers. Not only do you know what the output from the pooling layer for each example in the batch was, but you can look at the preceding layer and determine which input to the pool was the maximum.

Mathematically, and avoiding the need to define indices for NN layers and neurons, the rule can be expressed like this

  • The forward function is $m = max(a,b)$

  • We know $\frac{\partial J}{\partial m}$ for some target function J (in the neural network that will be the loss function we want to minimise, and we are assuming we have backpropagated to this point already)

  • We want to know $\frac{\partial J}{\partial a}$ and $\frac{\partial J}{\partial b}$

  • If $a > b$

    • Locally,* $m = a$. So $\frac{\partial J}{\partial a} = \frac{\partial J}{\partial m}$

    • Locally,* $m$ does not depend on $b$. So $\frac{\partial J}{\partial b} = 0$

  • Therefore $\frac{\partial J}{\partial a} = \frac{\partial J}{\partial m}$ if $a > b$, else $\frac{\partial J}{\partial a} = 0$

  • and $\frac{\partial J}{\partial b} = \frac{\partial J}{\partial m}$ if $b > a$, else $\frac{\partial J}{\partial b} = 0$

When back propagation goes across a max pooling layer, the gradient is processed per example and assigned only to the input from the previous layer that was the maximum. Other inputs get zero gradient. When this is batched it is no different, it is just processed per example, maybe in parallel. Across a whole batch this can mean that more than one, maybe all, of the input activations to the max pool get some share of the gradient - each from a different subset of examples in the batch.

* Locally -> when making only infinitesimal changes to $m$.

** Technically, if $a=b$ exactly then we have a discontinuity, but in practice we can ignore that without issues when training a neural network.

Improve this answer
$\endgroup$
  • 5
    $\begingroup$ Not sure I understand you. What you're saying works when you backprop after a single forward prop. But when you have a batch, you prop 100 samples, to calculate an average error function. The whole point of a batch is to be able to produce more accurate gradients, and then you only do once the backprop based on the derivative of the error function with respect to W, at the average error value found multiplied by the learning rate. So it's still no clear how you calc the derivate of the max function, when each time a different node was selected as a maximum. What am I missing? $\endgroup$ – Nadav B May 17 '17 at 12:39
  • 5
    $\begingroup$ @NadavB: You are missing the sequence of events. 1) The batch is calculated forward, item by item. 2) The batch is backpropagated, item by item. 3) Take the averages for gradients. 4) Apply a step of gradient descent (perhaps modified by something like Adagrad or RMSProp). So you are putting step 3 out of sequence and wondering how to unpick the average gradients over the max pool layer - but you never have to do this, because you backprop each example individually - you only aggregate results for the batch after that $\endgroup$ – Neil Slater May 17 '17 at 12:52
  • 1
    $\begingroup$ Perfect. You made it clear now. $\endgroup$ – Nadav B May 18 '17 at 10:39
  • $\begingroup$ @NeilSlater Is the backpropagation only done item by item because it is necessary for the max pool layer? In an MLP first averaging the error of the entire batch and then calculating the gradient on that average error is identical to calculating the gradient per item and then adjusting the parameters by the average gradient*learning rate, right? However, backpropagating the average error is much faster than backpropagating all individual errors and then applying the updates. So, when possible you would want to do it like this and only per item if necessary... like for max pool. Is that right? $\endgroup$ – lo tolmencre Jul 25 '19 at 15:35
  • $\begingroup$ @lotolmencre You are wrong about calculating the average error on a batch. You should back propagate individually, then sum gradients at the end. The gradient calculations will not work correctly otherwise through any nonlinearities. If you want to know more about that, please ask a new question $\endgroup$ – Neil Slater Jul 25 '19 at 16:01
2
$\begingroup$

I have same question, but I probably figure it out by reviewing source code of Caffe.

Please see source code of Caffe:

line 620 & 631 of this code.

It calculates derivative of each parameter by adding derivative (of this parameter ) of each input then divided it by batch size.

Also see line 137 of this code, it simply scales derivative to 1/iter_size, just same as average.

We can see there is NO special treatment for Max Pooling layer when BP.

As to derivative of Max Pooling , let's see source code of Caffe again:

line 272 of this code. Obviously, only the biggest element's derivative is 1*top_diff, others' derivative is 0*top_diff.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.