# Why do we need activation function (like ReLU) after an affine layer?

In Convolutional Neural Networks, assume the input and the output of the affine layer are $x$ and $y$, respectively. This affine operation $y = W^{\top} x + b$ has already add non-linearity to the system given that $b \neq 0$.

Why do we still need a function like ReLU to add non-linearity to the system?

## 1 Answer

This affine operation $y = W^{\top} x + b$ has already add nonlinearity to the system given that $b \neq 0$.

This is not considered a non-linearity in the context of data science. Different disciplines define linearity sometimes in subtly different ways. Critically, the $+b$ performs identically in terms of fitting to data, as extending $x$ with a new dimension, always $1$, and moving the values of $b$ into the weights $W$. This simpler multiplication is clearly linear.

Also importantly, the affine transformations form a group such that any two affine transformations combined are just another affine transformation with different parameters. Without a non-linearity in a hidden layer, a 2-layer neural network would be the same as a single layer one, and not able to learn whole classes on non-linear relations.

No matter how many affine transformations you apply to inputs for instance, you will not be able to approximate the XOR function, or any significant portion of $y=\text{sin}(x)$

• Thanks Neil! Strictly speaking, linear and affine are different in mathematics. Now I know that in data science, affine and linear can be viewed as equal in this case. Yes you are right, if the function is linear or affine, its combination also can be viewed as one linear or affine function. Thus we need a non-linear activation function like ReLU.
– J L
Jul 24 '18 at 12:52