What is the difference between "equivariant to translation" and "invariant to translation"

Question

I'm having trouble understanding the difference between equivariant to translation and invariant to translation.

In the book Deep Learning. MIT Press, 2016 (I. Goodfellow, A. Courville, and Y. Bengio), one can find on the convolutional networks:

• [...] the particular form of parameter sharing causes the layer to have a property called equivariance to translation
• [...] pooling helps to make the representation become approximately invariant to small translations of the input

Is there any difference between them or are the terms interchangeably used?

Answer 1

Equivariance and invariance are sometimes used interchangeably in common speech. They have ancient roots in maths and physics. As pointed out by @Xi'an, you can find previous uses (anterior to Convolutional Neural Networks) in the statistical literature, for instance on the notions of the invariant estimator and especially the Pitman estimator.

However, I would like to mention that it would be better if both terms keep separate meaning, as the prefix "in-" in invariant is privative (meaning "no variance" at all), while "equi-" in equivariant refers to "varying in a similar or equivalent proportion". In other words, one in- does not vary, the other equi- does.

Let us start from simple image features, and suppose that image $I$ has a unique maximum $m$ at spatial pixel location $(x_m,y_m)$, which is here the main classification feature. In other words: an image and all its translations are "the same". An interesting property of classifiers is their ability to classify in the same manner some distorted versions $I'$ of $I$, for instance translations by all vectors $(u,v)$.

The maximum value $m'$ of $I'$ is invariant: $m'=m$: the value is the same. While its location will be at $(x'_m,y'_m)=(x_m-u,y_m-v)$, and is equivariant, meaning that is varies "equally" with the distortion.

The precise formulations given (in mathematical terms) for equivariance depend on the class of objects and transformations one considers: translation, rotation, scale, shear, shift, etc. So I prefer here to focus on the notion that is most often used in practice (I accept the blame from a theoretical stand-point).

Here, translations by vectors $(u,v)$ of the image (or some more generic actions) can be equipped with a structure of composition, like that of a group $G$ (here the group of translations). One specific $g$ denotes a specific element of the translation group (translational symmetry). A function or feature $f$ is invariant under the group of actions $G$ if for all images in a class, and for any $g$, $$f(g(I)) = f(I)\,.$$

In other words: if you change the image by action $g$, the values for feature or function $f$ are the same.

It becomes equivariant if there exists another mathematical structure or action (often a group again) $G'$ that reflects the transformations (from $G$) in $I$ in a meaningful way. In other words, such that for each $g$, you have some (unique?) $g' \in G'$ such that

$$f(g(I)) = g'(f(I))\,.$$

In the above example on the group of translations, $g$ and $g'$ are the same (and hence $G'=G$): an integer translation of the image reflects as the exact same translation of the maximum location. This is sometimes refered to as "same-equivariance".

Another common definition is:

$$f(g(I)) = g(f(I))\,.$$

I however used potentially different $G$ and $G'$ because sometimes $f(.)$ and $g(.)$ do not lie in the same domain. This happens for instance in multivariate statistics (see e.g. Equivariance and invariance properties of multivariate quantile and related functions, and the role of standardisation). But here, the uniqueness of the mapping between $g$ and $g'$ allows to get back to the original transformation $g$.

Often, people use the term invariance because the equivariance concept is unknown, or everybody else uses invariance, and equivariance would seem more pedantic.

For the record, other related notions (esp. in maths and physics) are termed covariance, contravariance, differential invariance.

In addition, translation-invariance, as least approximate, or in envelope, has been a quest for several signal and image processing tools. Notably, multi-rate (filter-banks) and multi-scale (wavelets or pyramids) transformations have been design in the past 25 years, for instance under the hood of shift-invariant, cycle-spinning, stationary, complex, dual-tree wavelet transforms (for a review on 2D wavelets, A panorama on multiscale geometric representations). The wavelets can absorb a few discrete scale variations. All theses (approximate) invariances often come with the price of redundancy in the number of transformed coefficients. But they are more likely to yield shift-invariant, or shift-equivariant features.

Answer 2

The terms are different:

• Equivariant to translation means that a translation of input features results in an equivalent translation of outputs. So if your pattern 0,3,2,0,0 on the input results in 0,1,0,0 in the output, then the pattern 0,0,3,2,0 might lead to 0,0,1,0

• Invariant to translation means that a translation of input features doe not change the outputs at all. So if your pattern 0,3,2,0,0 on the input results in 0,1,0 in the output, then the pattern 0,0,3,2,0 would also lead to 0,1,0

For feature maps in convolutional networks to be useful, they typically need both properties in some balance. The equivariance allows the network to generalise edge, texture, shape detection in different locations. The invariance allows precise location of the detected features to matter less. These are two complementary types of generalisation for many image processing tasks.

Answer 3

Complementary to the previous answers - an image often says more than a thousand formulas.

Source: AMMI Seminar - Geometric Deep Learning and Reinforcement Learning (2021)

Answer 4

Just adding my 2 cents

Regarding an image classification task solved with a typical CNN Architecture consisting of a Backend (Convolutions + NL + possibly Spatial Pooling) which performs Representation Learning and of a Frontend (e.g. Fully Connected Layers, MLP) which solves the specific task, in this case image classification, the idea is to build a function $ f : I \rightarrow L $ able to map from the Spatial Domain $ I $ (Input Image) to the Semantic Domain $ L $ (Label Set) in a 2 step process which is

• Backend (Representation Learning) : $ f : I \rightarrow \mathcal{L} $ maps the Input to the Latent Semantic Space
• Frontend (Task Specific Solver) : $ f : \mathcal{L} \rightarrow L $ maps from the Latent Semantic Space to the Final Label Space

and it is performed using the following properties

• spatial equivariance, regarding ConvLayer (Spatial 2D Convolution+NonLin e.g. ReLU) as a shift in the Layer Input produces a shift in the Layer Output (Note: it is about the Layer, not the single Convolution Operator)
• spatial invariance, regarding the Pooling Operator (e.g. Max Pooling passes over the max value in its receptive field regardless of its spatial position)

The closer to the input layer, the closer to the purely spatial domain $ I $ and the more important the spatial equivariance property which allows to build spatially equivariant hierarchical (increasingly) semantic representation

The closer to the frontend, the closer to the latent purely semantic domain $ \mathcal{L} $ and the more important the spatial invariance as the specific meaning of the image is desired to be independent from the spatial positions of the features

Using fully connected layers in the frontend makes the classifier sensitive to feature position at some extent, depending on the backend structure : the deeper it is and the more the translation invariant operator (Pooling) used

It has been shown in Quantifying Translation-Invariance in Convolutional Neural Networks that to improve the CNN Classifier Translation Invariance, instead of acting on the inductive bias (architecture hence depth, pooling, …) it's more effective to act on the dataset bias (data augmentation)

Answer 5

These answers are kind of unintuitive so let me add my take from the mathematics point of view (should hopefully be clearer).

First recall the following concepts from Elementary Group Theory:

If $G$ is a group with identity $e$ and $X$ is a set then the left action $\alpha$ of $G$ on $X$ is a function $\alpha: G \times X \to X$ that satisfies: two properties:

1. for $x \in X$, we have $\alpha(e,x) = x$
2. $\alpha(g, \alpha(h, x)) = \alpha(gh, x)$ for all $g,h \in G$

Intuitively we can say then a group action $\alpha$ of $G$ on $X$ is a group homomorphism from $G \to Sym(X)$ of all bijections from $X$ to itself

Then recall:

• A G-set is a set with continuous group action $\mu: G \times X \to X$ where $X$ has the discrete topology

Then we get to the good stuff:

Equivariance:

• a form of symmetry for functions from one space with symmetry to another. More formally:

• A function is called \underline{equivariant} when it's domain and co-domain are acted on by the same symmetry group and when the function commutes with the action of the group. In essence: Let $G$ be a group, let $X,Y$ be G-sets, then $f: X \to Y$ is equivariant if $f(g \cdot x) = g \cdot f(x)$ for all $g \in G$ and $x \in X$

• In dumbed down terms: equivariant just means unaffected by a specific group action

Now note that: Equivariance generalizes invariance

Invariance: When a mathematical object is unchanged after operations or transformations of a certain type are applied to the object e.x. determinant and eigenvectors are invariant under change of basis

Citations:

[1] https://en.wikipedia.org/wiki/Equivariant_map

[2] https://en.wikipedia.org/wiki/Group_action

[3] https://ncatlab.org/nlab/show/G-set#:~:text=Given%20a%20topological%20group%20G,of%20the%20discrete%20group%20G.

[4] https://en.wikipedia.org/wiki/Invariant_(mathematics)