1
$\begingroup$

As I understand it, the point of architecting multiple layers in a neural network is so that you can have non-linearity represented in your deep network.

For example, this answer says: "To learn non-linear decision boundaries when classifying the output, multiple neurons are required."

When I watch online tutorials and whatnot, I see networks described as in the screenshot below. In cases like this, I see a series of linear classifiers:

enter image description here

We have a multiply, add, ReLu, multiply and add, all in series.

From studying math, I know that a composite function made out of linear functions is itself linear.

So how do you coax non-linearity out of multiple linear functions?

|improve this question
$\endgroup$
  • 1
    $\begingroup$ ReLUs are merely piecewise-linear. $\endgroup$ – Emre Sep 16 '16 at 4:35
  • $\begingroup$ Exactly! So how would using a ReLu along with other linear operators allow you to access nonlinearity in a neural network? $\endgroup$ – Monica Heddneck Sep 16 '16 at 4:39
  • $\begingroup$ @Emre already answered you. Piecewise-linear means it's only linear in segments. $\endgroup$ – HelloWorld Sep 16 '16 at 5:05
  • $\begingroup$ So the statement "To learn non-linear decision boundaries when classifying the output, multiple neurons are required." is incorrect? $\endgroup$ – Monica Heddneck Sep 16 '16 at 5:41
3
$\begingroup$

The phase

"To learn non-linear decision boundaries when classifying the output, multiple neurons are required."

is NOT correct. More precisely, it should be:

"To learn non-linear decision boundaries when classifying the output, we need a non-linear activation function."

To understand why, imagine you have a network with many layers and nodes (multiple neurons in your question). If you don't have a non-linear activation function such as ReLu or sigmoid, your network is just a linear combination of bias and weights. Your network won't be useful for classifying non-linear decision boundary. But if your inputs can be linearly separable, you don't need neutral network...

That's why all neutral networks almost always have a non-linear activation function. ReLu is the most popular, but there are other possibilites. When you pipe up a dozen of non-linear outputs like in neutral network, your network will be able to classify a non-linear decision boundary. The more your have, the better it can perform (but also easier for overfitting).

|improve this answer
$\endgroup$
  • $\begingroup$ Great answer. Would you agree with the answer to this question -- that we can combine ReLus to approximate nonlinear functions? stats.stackexchange.com/questions/141960/… $\endgroup$ – Monica Heddneck Sep 16 '16 at 7:50
  • $\begingroup$ @MonicaHeddneck The answer there is correct, and is really what I said without the mathematics details (pipe up a dozen of non-linear outputs). $\endgroup$ – HelloWorld Sep 16 '16 at 7:53
  • $\begingroup$ Wow. That's amazing and I've learned so much. How would you know what type of these more complicated ReLus you'd need...or what order to put them in? $\endgroup$ – Monica Heddneck Sep 16 '16 at 7:54
  • $\begingroup$ @MonicaHeddneck That's parameter optimisation. Nobody knows for sure, we'll need to train the model and evaluate. ML is hard... $\endgroup$ – HelloWorld Sep 16 '16 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.