ReLU is defined as being $x \mapsto x$ whenever $x \geq 0$ and is constant on zero for negative numbers.
I'm a beginner to deep learning research and methodologies but I've already seen several examples that claim that using ReLU as activation functions on a network will be superior than other common functions, including simply linear ones, of course.
ReLU has 1 point of "drastic" change, and is "otherwise linear".
My questions are:
Is there research or opinions on why this change significantly affects the performance of a network?
Is there something unique about $0$, the point of change? What if we move the point of change to $1$ instead for instance?
My intuition is that, as $0$ separates the line, it separates the sign function and that may have high affect on certain types of data that rely on positive/negative classification, perhaps.
- How relevant is the dataset being learned? Do we see that ReLU is "better" on most examples of datasets?
The above question is relevant. Mine is more focused on the difference between ReLU and a linear function.