# How do you make an NN for image classification invariant to translation **and** rotation?

CNN are (I think) invariant to small translations of the input image, i.e., they will classify to the same class an image $X$ and an image $X'$ such that all pixels have been translated along a vector $\mathbf{v}$ by some "small" distance $d$. They're not, however, invariant to rotation. I.e., if $X'$ is obtained by $X$ applying a rotation of arbitrary degree $\theta$ around an axis $z$ orthogonal to the plane of the image, they won't necessarily classify it to the same class as $X$. Practitioners usually solve this by data augmentation, but this is unnecessarily wasteful and anyway not an option in my case. Is there a way to make the NN architecture invariant to both isometries? I don't have to use a CNN, but since CNN already enjoy some sort of translation invariance, I figured it would be easier to modify them to get rotational invariance , than to use a completely different architecture.

It is not possible to have general rotationally-invariant neural network architecture for a CNN*. In fact CNNs are not strongly translation invariant, except due to pooling - instead they combine a little bit of translation invariance with translation equivariance. There is no equivalent to pooling layers that would reduce the effect of rotation this way (although for very small rotations the translation invariance will still help).

You can however construct features and create a pipeline that reduces incoming rotation differences in your inputs. For example, this answer on Signal Processing Stack Exchange suggests calculating dominant gradient of an image, then rotating the image so that this is always oriented the same way before further processing. If your image has strong straight edges, you could do similar by detecting those (e.g. by using a Hough transform) and rotating the input so that these are always oriented the same way. These approaches work only within certain image tasks, but can save on processing time and potentially increase accuracy if they are possible. Effectively they are a form of input normalisation.

A more radical idea might be to perform a map to polar co-ordinates in your image before processing. This would effectively convert rotational (and radial) variance into translation variance in your image. A CNN processing this mapped image would effectively convert its translation invariance into rotational invariance on the original unmapped image. But the cost would be losing all translation invariance, so only worth considering if your inputs have high variance in rotation but low variance in translation.

* Never say never. This is caveated by special cases, for example the paper Group Equivariant Convolutional Networks explains an architecture that adds support for multiples of 90 degree rotation (taking advantage of grid structure in computer images and the weight matrices that construct neural network layers). However, if you want to support free rotation of values other than 90, 180, 270 degrees, then as far as I know, there is no way to do that architecturally within the network.

• I would like to add that there are convolutional architectures that have specific rotation equivariances, for example 90, 180 and 270 degrees. See: arxiv.org/abs/1602.07576 Jul 5 '17 at 9:30
• I didn't know about the equivarance concept: I'll study your link. Will also look into Hough transform. Using equivariance as a search word, I found this: arxiv.org/pdf/1612.04642.pdf It's very complex for me. It seems to preserve both translational & rotational equivarance . What do you think, will it work? I can open a new question if you want. Jul 5 '17 at 11:20
• @DeltaIV: The paper is too advanced for me, too, but might help you with your problem. Perhaps someone else could answer here using details from the paper, or if you dig into the paper and get stuck. One thing to note - equivariance can help with reducing number of parameters required (it's essentially a core concept to explain why CNNs perform well), but it doesn't always help with every problem. Jul 5 '17 at 11:55

Please see 3 recent pre-prints (6-8/2018) about transformation-identical CNN (TI-CNN) and geared rotation-identical CNN (GRI-CNN). They were expanded and derived from the circular path CNN (SPIE MI 1998 and IEEE TMI 2002) initiated by the same medical AI research team at Georgetown University medical center who also independently developed the CNN and the wavelet CNN in early 1990s for cancer research.

“Transformationally Identical and Invariant Convolutional Neural Networks through Symmetric Element Operators” https://arxiv.org/abs/1806.03636

“Transformationally Identical and Invariant Convolutional Neural Networks by Combining Symmetric Operations or Input Vectors” https://arxiv.org/abs/1807.11156

"Geared Rotationally Identical and Invariant Convolutional Neural Network Systems" https://arxiv.org/abs/1808.01280 also