# Neural network linearity and non linearity

Is it right on my part to call:

• A neural network with only input and output layer (sigmoid) as linear (since it is a logistic regression)

• A neural network with more than one hidden layer non-linear (since weights interact with outputs from hidden layer and there is no way to write output as op = w2x1+w2x2+b like in linear regression)?

Please don't mark this as duplicate as none of the questions actually answer the question asked here. Thank you.

• What if you have linear activation functions in the hidden layer? // What if you do a standard MLP but no activation function in the output node?
– Dave
Jun 18, 2021 at 17:06

Yes you are mostly correct. A feedforward neural network with a single layer and a sigmoid activation is a logistic regression which belongs to GLM type of models. Your second statement is unclear (weights interact with outputs) so I will try to break this down below:

Non-linear transformations (e.g. polynomial regression, logistic unit etc.) is often misread for non-linearity in model parameters (non-linear models).

As an example let's look at a feedforward neural network architecture. For $$f(x)$$ activation function and $$w,b$$ weights and biases, the output of a neuron from the first layer of a feed forward network would look like:

$$a_{11}=f(w_{11}x_1 + w_{12}x_2 + .. +b_1)$$

whereas the output of a neuron from the second layer neural of that neural network would look like:

$$a_{21}=f(w_{21}a_{11}+ w_{22}a_{22} + .. + b_2)$$, and given $$a_1$$ above $$\Rightarrow a_2=f(w_2w_1x+w_2b1+b2)$$

$$a_{21}=w_{21}w_{11}x_1 + w_{21}w_{12}x_2+ .. + b_2$$

The multiplication between parameters (here $$w_1w_2$$) is what makes a model non-linear. In order to acquire that you need:

• Either multiple layers OR
• A non-linearity from the activation function e.g. if $$f(x)=x^2$$ even the output from the first layer of the neural network would be $$a_{11}=w_{11}^2x^2 + b_2^2+w_{11}b_1x$$ that qualifies as parameter multiplication in the $$w_{11}^2$$ factor and a non-linear model of a non-linear relationship with additive errors.

Hope it helps.

Sauces:

• "Single-layer" is confusing. Do you mean a network with one hidden layer or zero? // Even a network with no hidden layers does not have to be a logistic regression. It could be a linear model or any other kind of generalized linear model (or perhaps even a weirder activation function than GLMs use as link functions).
– Dave
Jun 21, 2021 at 13:15
• Single layer neural network, single layer perceptron are widely used in the literature afaik, e.g.en.wikipedia.org/wiki/…. On your second point you are right, a sigmoid activation needs to be in place so that it is a logistic regression - i have amended this mistake now. Jun 21, 2021 at 14:22