# If ReLU is so close to being linear, why does it perform much better than a linear function?

ReLU is defined as being $$x \mapsto x$$ whenever $$x \geq 0$$ and is constant on zero for negative numbers.

I'm a beginner to deep learning research and methodologies but I've already seen several examples that claim that using ReLU as activation functions on a network will be superior than other common functions, including simply linear ones, of course.

ReLU has 1 point of "drastic" change, and is "otherwise linear".

My questions are:

1. Is there research or opinions on why this change significantly affects the performance of a network?

2. Is there something unique about $$0$$, the point of change? What if we move the point of change to $$1$$ instead for instance?

My intuition is that, as $$0$$ separates the line, it separates the sign function and that may have high affect on certain types of data that rely on positive/negative classification, perhaps.

1. How relevant is the dataset being learned? Do we see that ReLU is "better" on most examples of datasets?

Why ReLU is better than the other activation functions

The above question is relevant. Mine is more focused on the difference between ReLU and a linear function.

1. A neural network is a succession of layers (dense/linear, convolutional). After each layer, you need a non-linear activation function. Why do you need a non-linear activation function? Because composing two linear transformations is equivalent to having a single linear transformation:

$$(x \cdot W_1 + b_1) \cdot W_2 + b_2 = x (W_1 \cdot W_2) + (b_1 \cdot W_2 + b_2)$$

Therefore, stacking linear transformations would be pointless, because the whole network would be equivalent to a linear transformation.

A ReLU may seem similar to a linear function, but its non-linear nature gives the network the ability to model non-linear functions.

2. No, there is nothing unique about $$0$$ being the point of change. Most neural network layers have a bias term that makes the point of change irrelevant, as during the training the bias will be adjusted to the proper value.

3. While in deep learning everything is data-dependent, the fact that the ReLU's changing point is irrelevant is independent from the data (see previous answer).

The advantage of ReLU is against other non-linearities, like sigmoid and tanh, which suffer the vanishing gradient problem. ReLU itself has its own problem, namely the "dying ReLU problem".

• Thanks! Do you have some references about the second clause? I'd like to better understand how this affects the point of change. – Mariah Feb 26 '20 at 16:24
• I don't remember having seen any discussion about this. My answer (2) is pure deduction: you have a trainable free term (bias) added to the output of the operation (matrix multiplication / convolution) before the activation. Modifying the "changing point" of the ReLU is like subtracting a scalar. The bias would certainly compensate for any scalar you subtract. – noe Feb 26 '20 at 16:44
• How is it like subtracting a scalar? If you subtract $a > 0$ then the graph "falls" $a$ units. In order to move the changing point you need to do a variable change $x \mapsto x + a$ to move it left $a$ units. – Mariah Feb 26 '20 at 17:03
• Sorry, I meant "adding" instead of "subtracting", and that's precisely what you wrote. These $a$ would be easily compensated by the learned bias. – noe Feb 26 '20 at 17:42

In addition to the ncasas' answer, which is good in my opinion, I'd like to point out that ReLU is computationally inexpensive, in contrast to sigmoid activation functions. They require only an if / then comparison, while e.g. the logistic function requires exponentiation, addition, and division. This practical consideration makes ReLU's attractive, especially when the computation is to be performed on simpler processing units, like the GPUs.

A ReLU serves as a non-linear activation function. If a network had a linear activation function, then it wouldn't be able map any non-linear relationships between the input features and its targets. This would render all hidden layers redundant, as your model would just be a much more complex logistic regression.

So I think a better question you should be asking is: Why do you need a non-linear activation function?

As an intuitive example on what this means you can head on to TensorFlow playground and try this out for yourself. Just change the activation function; the rest of the default settings work well to illustrate the point.

• URL for trial with ReLU
• URL for the same function with a linear activation.
• Thanks for your answer Djib. I understand the need to map non-linear relationships, but that is not what my question is about. I'm interested in the particular properties of ReLU, see my question. – Mariah Feb 26 '20 at 9:24
• You're asking why ReLU and not a linear activation, if I'm not mistaken. The point is that ReLU is not-linear. Regarding ReLU vs the other non-linear activation functions, the answer you cited should cover you. – Djib2011 Feb 26 '20 at 10:30