I understand the advantages of ReLU, which is avoiding dead neurons during backpropagation.
This is not completely true. The neurons are not dead. If you use sigmoid-like activations, after some iterations the value of gradients saturate for most the neurons. The value of gradient will be so small and the process of learning happens so slowly. This is vanishing and exploding gradients that has been in sigmoid-like activation functions. Conversely, the dead neurons may happen if you use
ReLU non-linarity, which is called dying ReLU.
I am not able to understand why is ReLU used as an activation function if its output is linear
Definitely it is not linear. As a simple definition, linear function is a function which has same derivative for the inputs in its domain.
The linear function is popular in economics. It is attractive because it is simple and easy to handle mathematically. It has many important applications. Linear functions are those whose graph is a straight line. A linear function has the following form:
y = f(x) = a + bx
A linear function has one independent variable and one dependent variable. The independent variable is x and the dependent variable is y.
a is the constant term or the y intercept. It is the value of the dependent variable when x = 0.
b is the coefficient of the independent variable. It is also known as the slope and gives the rate of change of the dependent variable.
ReLU is not linear. The simple answer is that
ReLU output is not a straight line, it bends at the x-axis. The more interesting point is what’s the consequence of this non-linearity. In simple terms, linear functions allow you to dissect the feature plane using a straight line. But with the non-linearity of
ReLUs, you can build arbitrary shaped curves on the feature plane.
ReLU may have a disadvantage which is its expected value. There is no limitation for the output of the
Relu and its expected value is not zero.
Tanh was more popular than
sigmoid because its expected value is equal to zero and learning in deeper layers occurs more rapidly. Although
ReLU does not have this advantage
batch normalization solves this problem.
You can also refer here and here for more information.