In Convolutional Neural Networks, assume the input and the output of the affine layer are $x$ and $y$, respectively. This affine operation $y = W^{\top} x + b$ has already add non-linearity to the system given that $b \neq 0$.

Why do we still need a function like ReLU to add non-linearity to the system?


This affine operation $y = W^{\top} x + b$ has already add nonlinearity to the system given that $b \neq 0$.

This is not considered a non-linearity in the context of data science. Different disciplines define linearity sometimes in subtly different ways. Critically, the $+b$ performs identically in terms of fitting to data, as extending $x$ with a new dimension, always $1$, and moving the values of $b$ into the weights $W$. This simpler multiplication is clearly linear.

Also importantly, the affine transformations form a group such that any two affine transformations combined are just another affine transformation with different parameters. Without a non-linearity in a hidden layer, a 2-layer neural network would be the same as a single layer one, and not able to learn whole classes on non-linear relations.

No matter how many affine transformations you apply to inputs for instance, you will not be able to approximate the XOR function, or any significant portion of $y=\text{sin}(x)$

  • $\begingroup$ Thanks Neil! Strictly speaking, linear and affine are different in mathematics. Now I know that in data science, affine and linear can be viewed as equal in this case. Yes you are right, if the function is linear or affine, its combination also can be viewed as one linear or affine function. Thus we need a non-linear activation function like ReLU. $\endgroup$
    – J L
    Jul 24 '18 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.