I have been reading a paper recently on Highway Neural Networks and found the following:


$H$ is usually an affine transform followed by a non-linear activation function, but in general it may take other forms.

After googling about affine transform I can't say I understand fully what it means. Can somebody please elaborate?


It's a linear transformation, for example lines that were parallel before the transformation are still parallel. Scaling, rotation, reflection etcetera. With regards to neural networks it's usually just the input matrix multiplied by the weight matrix.


Affine transformation is of the form, $$ g(\vec(v) = Av+b $$ where, $A$ is the matrix representing a linear transformation and $b$ is a vector.

In other words, affine transformation is the combination of linear transformation with translation.

Linear transformation always carry vector $b$ = 0 in the source space to 0 in target space.


$ y=3x + 4$ , in school we called it linear equation, but it is not speaking strictly about linear transformation, because it has translation (+4), and linear transformation don't do that.

so, every linear transformation is affine (just set b to the zero vector). However, not every affine transformation is linear.

Now, in context of machine learning, linear regression attempts to fit a line on to data in an optimal way,

line being defined as , $ y=mx+b$. As explained its not actually a linear function its an affine function. And probably should be renamed. Its good to get the terminologies right.

Similarly, in a single layer of neural network is often expressed mathematically as: $$y(\vec{x})=W\vec{x}+\vec{b} $$

$W$ is the weight matrix and $\vec{b}$ is the bias vector. This function is also usually referred to as linear although it's actually affine.


If $x$ is your input vector, an affine transformation over $x$ will have this form:

$$ y = Ax+b $$

where the coefficients of the matrix $A$ and the vector $b$ are the parameters of the transformation. So it's like a linear function over $x$ but it's not a linear mapping in terms of vector spaces, although you can take it to the form of a linear mapping using homogeneous coordinates.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.