1
$\begingroup$

As I understand it, the point of architecting multiple layers in a neural network is so that you can have non-linearity represented in your deep network.

For example, this answer says: "To learn non-linear decision boundaries when classifying the output, multiple neurons are required."

When I watch online tutorials and whatnot, I see networks described as in the screenshot below. In cases like this, I see a series of linear classifiers:

enter image description here

We have a multiply, add, ReLu, multiply and add, all in series.

From studying math, I know that a composite function made out of linear functions is itself linear.

So how do you coax non-linearity out of multiple linear functions?

$\endgroup$
4
  • 1
    $\begingroup$ ReLUs are merely piecewise-linear. $\endgroup$
    – Emre
    Sep 16, 2016 at 4:35
  • $\begingroup$ Exactly! So how would using a ReLu along with other linear operators allow you to access nonlinearity in a neural network? $\endgroup$ Sep 16, 2016 at 4:39
  • $\begingroup$ @Emre already answered you. Piecewise-linear means it's only linear in segments. $\endgroup$
    – SmallChess
    Sep 16, 2016 at 5:05
  • $\begingroup$ So the statement "To learn non-linear decision boundaries when classifying the output, multiple neurons are required." is incorrect? $\endgroup$ Sep 16, 2016 at 5:41

1 Answer 1

3
$\begingroup$

The phase

"To learn non-linear decision boundaries when classifying the output, multiple neurons are required."

is NOT correct. More precisely, it should be:

"To learn non-linear decision boundaries when classifying the output, we need a non-linear activation function."

To understand why, imagine you have a network with many layers and nodes (multiple neurons in your question). If you don't have a non-linear activation function such as ReLu or sigmoid, your network is just a linear combination of bias and weights. Your network won't be useful for classifying non-linear decision boundary. But if your inputs can be linearly separable, you don't need neutral network...

That's why all neutral networks almost always have a non-linear activation function. ReLu is the most popular, but there are other possibilites. When you pipe up a dozen of non-linear outputs like in neutral network, your network will be able to classify a non-linear decision boundary. The more your have, the better it can perform (but also easier for overfitting).

$\endgroup$
4
  • $\begingroup$ Great answer. Would you agree with the answer to this question -- that we can combine ReLus to approximate nonlinear functions? stats.stackexchange.com/questions/141960/… $\endgroup$ Sep 16, 2016 at 7:50
  • $\begingroup$ @MonicaHeddneck The answer there is correct, and is really what I said without the mathematics details (pipe up a dozen of non-linear outputs). $\endgroup$
    – SmallChess
    Sep 16, 2016 at 7:53
  • $\begingroup$ Wow. That's amazing and I've learned so much. How would you know what type of these more complicated ReLus you'd need...or what order to put them in? $\endgroup$ Sep 16, 2016 at 7:54
  • $\begingroup$ @MonicaHeddneck That's parameter optimisation. Nobody knows for sure, we'll need to train the model and evaluate. ML is hard... $\endgroup$
    – SmallChess
    Sep 16, 2016 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.