# What is the difference between “equivariant to translation” and “invariant to translation”

I'm having trouble understanding the difference between equivariant to translation and invariant to translation.

In the book Deep Learning. MIT Press, 2016 (I. Goodfellow, A. Courville, and Y. Bengio), one can find on the convolutional networks:

• [...] the particular form of parameter sharing causes the layer to have a property called equivariance to translation
• [...] pooling helps to make the representation become approximately invariant to small translations of the input

Is there any difference between them or are the terms interchangeably used?

• In the old days of Statistics, as in the time of Pitman, invariant was used in the meaning of equivariant. – Xi'an Oct 12 '18 at 18:12

Equivariance and invariance are sometimes used interchangeably in common speech. They have ancient roots in maths and physics. As pointed out by @Xi'an, you can find previous uses (anterior to Convolutional Neural Networks) in the statistical literature, for instance on the notions of the invariant estimator and especially the Pitman estimator.

However, I would like to mention that it would be better if both terms keep separate meaning, as the prefix "in-" in invariant is privative (meaning "no variance" at all), while "equi-" in equivariant refers to "varying in a similar or equivalent proportion". In other words, one in- does not vary, the other equi- does.

Let us start from simple image features, and suppose that image $$I$$ has a unique maximum $$m$$ at spatial pixel location $$(x_m,y_m)$$, which is here the main classification feature. In other words: an image and all its translations are "the same". An interesting property of classifiers is their ability to classify in the same manner some distorted versions $$I'$$ of $$I$$, for instance translations by all vectors $$(u,v)$$.

The maximum value $$m'$$ of $$I'$$ is invariant: $$m'=m$$: the value is the same. While its location will be at $$(x'_m,y'_m)=(x_m-u,y_m-v)$$, and is equivariant, meaning that is varies "equally" with the distortion.

The precise formulations given (in mathematical terms) for equivariance depend on the class of objects and transformations one considers: translation, rotation, scale, shear, shift, etc. So I prefer here to focus on the notion that is most often used in practice (I accept the blame from a theoretical stand-point).

Here, translations by vectors $$(u,v)$$ of the image (or some more generic actions) can be equipped with a structure of composition, like that of a group $$G$$ (here the group of translations). One specific $$g$$ denotes a specific element of the translation group (translational symmetry). A function or feature $$f$$ is invariant under the group of actions $$G$$ if for all images in a class, and for any $$g$$, $$f(g(I)) = f(I)\,.$$

In other words: if you change the image by action $$g$$, the values for feature or function $$f$$ are the same.

It becomes equivariant if there exists another mathematical structure or action (often a group again) $$G'$$ that reflects the transformations (from $$G$$) in $$I$$ in a meaningful way. In other words, such that for each $$g$$, you have some (unique?) $$g' \in G'$$ such that

$$f(g(I)) = g'(f(I))\,.$$

In the above example on the group of translations, $$g$$ and $$g'$$ are the same (and hence $$G'=G$$): an integer translation of the image reflects as the exact same translation of the maximum location. This is sometimes refered to as "same-equivariance".

Another common definition is:

$$f(g(I)) = g(f(I))\,.$$

I however used potentially different $$G$$ and $$G'$$ because sometimes $$f(.)$$ and $$g(.)$$ do not lie in the same domain. This happens for instance in multivariate statistics (see e.g. Equivariance and invariance properties of multivariate quantile and related functions, and the role of standardisation). But here, the uniqueness of the mapping between $$g$$ and $$g'$$ allows to get back to the original transformation $$g$$.

Often, people use the term invariance because the equivariance concept is unknown, or everybody else uses invariance, and equivariance would seem more pedantic.

For the record, other related notions (esp. in maths and physics) are termed covariance, contravariance, differential invariance.

In addition, translation-invariance, as least approximate, or in envelope, has been a quest for several signal and image processing tools. Notably, multi-rate (filter-banks) and multi-scale (wavelets or pyramids) transformations have been design in the past 25 years, for instance under the hood of shift-invariant, cycle-spinning, stationary, complex, dual-tree wavelet transforms (for a review on 2D wavelets, A panorama on multiscale geometric representations). The wavelets can absorb a few discrete scale variations. All theses (approximate) invariances often come with the price of redundancy in the number of transformed coefficients. But they are more likely to yield shift-invariant, or shift-equivariant features.

• Great! I really admire your effort for the detailed reply @Laurent Duval – Aamir Jan 5 '17 at 8:32
• The case where $g=g'$ has been dubbed same-equivariance and I think it is a very useful concept. Same-equivariance is much more tractable than equivariance. Furthermore, when people talk about equivariance, they usually mean it in this more narrow sense. – Pepe Mandioca Jan 9 '20 at 1:53
• The same-equivariant is, to me, the same as equivariant in its most mundane version, in that it varies "in the same manner" (and the one often unappropriately called invariant). – Laurent Duval Jan 11 '20 at 19:01
• Well, the same could be said in the case of invariance vs equivariance, that is, invariance is equivariance in its most mundane version (the $id$ function). In many cases, same-equivariance is important, because for example it usually means that $g$ can be applied to both the input and output and that gives certain structure that can be exploited. – Pepe Mandioca Feb 22 at 20:16

The terms are different:

• Equivariant to translation means that a translation of input features results in an equivalent translation of outputs. So if your pattern 0,3,2,0,0 on the input results in 0,1,0,0 in the output, then the pattern 0,0,3,2,0 might lead to 0,0,1,0

• Invariant to translation means that a translation of input features doe not change the outputs at all. So if your pattern 0,3,2,0,0 on the input results in 0,1,0 in the output, then the pattern 0,0,3,2,0 would also lead to 0,1,0

For feature maps in convolutional networks to be useful, they typically need both properties in some balance. The equivariance allows the network to generalise edge, texture, shape detection in different locations. The invariance allows precise location of the detected features to matter less. These are two complementary types of generalisation for many image processing tasks.

• Translated feature yields translated output at some layer. Please elaborate about considerably translated whole object being detected. Seems it will be detected even if CNN was not trained with images containing different positions? Does equivariance hold in this case (looks more similar to invariance)? – VladimirLenin Jul 14 '17 at 10:14
• @VladimirLenin: I don't think that elaboration is required for this question, it is definitely not something the OP has asked here. I suggest you ask a separate question, with a concrete example if possible. Even if visually a "whole object" has been translated, that does not mean feature maps in a CNN are tracking the same thing as you expect. – Neil Slater Jul 14 '17 at 10:24

Regarding an image classification task solved with a typical CNN Architecture consisting of a Backend (Convolutions + NL + possibly Spatial Pooling) which performs Representation Learning and of a Frontend (e.g. Fully Connected Layers, MLP) which solves the specific task, in this case image classification, the idea is to build a function $f : I \rightarrow L$ able to map from the Spatial Domain $I$ (Input Image) to the Semantic Domain $L$ (Label Set) in a 2 step process which is

• Backend (Representation Learning) : $f : I \rightarrow \mathcal{L}$ maps the Input to the Latent Semantic Space
• Frontend (Task Specific Solver) : $f : \mathcal{L} \rightarrow L$ maps from the Latent Semantic Space to the Final Label Space

and it is performed using the following properties

• spatial equivariance, regarding ConvLayer (Spatial 2D Convolution+NonLin e.g. ReLU) as a shift in the Layer Input produces a shift in the Layer Output (Note: it is about the Layer, not the single Convolution Operator)
• spatial invariance, regarding the Pooling Operator (e.g. Max Pooling passes over the max value in its receptive field regardless of its spatial position)

The closer to the input layer, the closer to the purely spatial domain $I$ and the more important the spatial equivariance property which allows to build spatially equivariant hierarchical (increasingly) semantic representation

The closer to the frontend, the closer to the latent purely semantic domain $\mathcal{L}$ and the more important the spatial invariance as the specific meaning of the image is desired to be independent from the spatial positions of the features

Using fully connected layers in the frontend makes the classifier sensitive to feature position at some extent, depending on the backend structure : the deeper it is and the more the translation invariant operator (Pooling) used

It has been shown in Quantifying Translation-Invariance in Convolutional Neural Networks that to improve the CNN Classifier Translation Invariance, instead of acting on the inductive bias (architecture hence depth, pooling, …) it's more effective to act on the dataset bias (data augmentation)