2
$\begingroup$

a nonlinear model of a neuron

TL;DR: in what sense is the model of a neuron seen in the image above nonlinear?

In chapter 1, section 1.3 MODELS OF A NEURON of Simon Haykin's Neural Networks book, the standard model of a single neuron is described and visualised in the picture above.

Haykin states that this model, which consists of a set of inputs x1..xm, their corresponding weights w1..wm, a linear combiner that sums the weighted inputs and the bias (b) and an activation function that takes that sum and produces the output, is nonlinear. So, my question is, isn't the output linearly dependent on the input? For example, if the neuron only takes one input, x1, then the linear combiner takes the form v = x1 + b and the activation function is φ(v). So, the only way that I can see this model being nonlinear is if the activation function is nonlinear. But there are clearly cases where the activation function is linear (like the piecewise-linear function described in that same section of the book). So how can this model be inherently nonlinear?

I realise that this isn't a major concern, but I'd like to understand every part of the book before moving on, and this has been bugging me since I saw it.

Thanks to everyone in advance for your answers.

$\endgroup$
1
$\begingroup$

You are right, for the model to be non linear, the activation function must be non linear.

If the activation function is for example the identity function, your model won't be non linear, even if you stack multiple hidden layers. Indeed, the output of the neurons will only just be a linear composition of the inputs combined with weights.

I think the author says this neuron model is non-linear because in practice a linear activation function is almost never used. Some functions look like linear functions, but in fact these are not. Here is a demonstration for the ReLU activation function, which look like a linear function but is mathematically not : https://datascience.stackexchange.com/a/26481/73209

I guess it is the same for the Piecewise Linear Unit (PLU) activation function.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for that sanity check. I'm still concerned, however, about why the author persistently states that this model of a neuron is "nonlinear". I find it difficult to believe it's just a simplification or oversight. $\endgroup$ – Alexgeo2395 Jul 31 '19 at 10:14
  • $\begingroup$ I edited my response with feedback $\endgroup$ – Alexis Pister Jul 31 '19 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.