2
$\begingroup$

What is the reasoning behind ReLu boundary lines appearing linear when plotting in 2D? Does this generalize to higher dimensions in that boundary lines in large dimensions are linear hyperplanes as well?

For instance, in tensorflow's playground any time ReLu is chosen as the activation function, the resulting boundary line is made of lines.

I'm failing to see the connection between the underlying neurons outputting only z or 0 and how that connects to the output plot.

Thanks very much for any thoughts/help!

Improve this question
$\endgroup$
3
  • $\begingroup$ Please elaborate more with an example. I am not able to comprehend your actual ask $\endgroup$
    – 10xAI
    May 15, 2020 at 9:26
  • $\begingroup$ If you follow this link and press the play button, the resulting decision boundary is made of linear lines. This is a consequence of the ReLu activation function, but I am uncertain as to why. $\endgroup$
    – user12346170
    Jun 2, 2020 at 22:51
  • $\begingroup$ See also datascience.stackexchange.com/q/76022/55122 $\endgroup$
    – Ben Reiniger ♦
    Apr 7 at 17:21

1 Answer 1

Reset to default
0
$\begingroup$

As far as i understand, relu gives linear boundary because it is linear for X>0. If you expand it to write it in the form of an equation like Z=f1(x1)+f2(x2)....fn(xn), we simply get a linear equation even if we consider the dead neurons. The part where x<0 makes sure that no -ve signals are carried forward but dosent add any non linearity to the final equation. The effect of dead neurons probaly the the final coefficients of the equaion will have coefficients with slightly less value than had the activation been just a normal fuction since there is no negative component which is added to a neuron.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.