1
$\begingroup$

I read that in a hidden layer of a convolutional neural network, all neurons share the same weights and bias. As a result, all the neurons detect the same feature and hence ConvNets become invariant to translation.

And then I read that in 2003 Simard, Steinkraus and Platt achieved better classification in MNIST classification by artificially expanding the data using translation among other operations.

So, my question is are ConvNets really invariant to translation?

|improve this question
$\endgroup$
  • $\begingroup$ Are you familiar with spatial transformer modules used in CNNs? $\endgroup$ – Media Oct 19 '19 at 21:15
3
$\begingroup$

Short answer - No, CNNs are not really translation invariant.

I specifically mean the style of image classification network in the paper you mentioned (i.e., (input) > (conv layer) > ... > (conv layer) > (fully conn layer) ... > (fully conn layer) > (output)).

This is partly because of the difference between translation invariance and translation equivariance (an important distinction, imo). See this question and the answers

Why

The last few layers in the network are fully connected (FC). FC layers are definitely not translation invariant (they don't give consideration to the spatial relationship between the inputs). But, the whole network could still be translation invariant if everything before the first FC layer is translation invariant.

We have convolution layers before the FC layers, but convolution layers are translation equivariant, not translation invariant (see below, it'll make sense). Therefore the whole network is not (really) translation invariant.

Adding pooling layers after convolution makes the network invariant to small translation motions.

Convolution operations are translation equivariant

Translation equivariance of a function means that it's output for a translated version of the input is a translated version of the output.

if $$f(x(i)) = y(i)$$ then $$f(x(i-t)) = y(i-t)$$ where $i$ is the spatial index

Compare to translation invariance

Translation invariance means that the output of a translated version of the intput, is exactly the same as the output for the original input.

if $$f(x(i)) = y(i)$$ then $$f(x(i-t)) = y(i)$$ where $i$ is the spatial index

|improve this answer
$\endgroup$
  • 1
    $\begingroup$ It's crystal clear. Can you add one more statement to your answer? "Adding pooling layers after convolution makes the network invariant to small translation motions". I wanted to ask this question, but found the answer in the question you tagged. $\endgroup$ – Nagabhushan S N Oct 6 '19 at 15:37
  • 1
    $\begingroup$ Good point. Added that bit aboutpooling layers. $\endgroup$ – bogovicj Oct 6 '19 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.