As described in the paper "The all convolutional net", pooling can be described with the following formula (see image). With p → ∞ (maximum norm) it becomes the max pooling operation.
Every value in the outcoming feature map is mapped to the maximum value of multiple values of the original feature map. The indexes of these values are calculated in the green marked formula.
How can the start and end value for shifting be the same for two different kernel sizes ?
(see purple marked formula)
|kernel size = 2||kernel size = 3|
|start index = -1||start index = -1|
|end index = 1||end index = 1|