1
$\begingroup$

Is it right on my part to call:

  • A neural network with only input and output layer (sigmoid) as linear (since it is a logistic regression)

  • A neural network with more than one hidden layer non-linear (since weights interact with outputs from hidden layer and there is no way to write output as op = w2x1+w2x2+b like in linear regression)?

Please don't mark this as duplicate as none of the questions actually answer the question asked here. Thank you.

$\endgroup$
1
  • 1
    $\begingroup$ What if you have linear activation functions in the hidden layer? // What if you do a standard MLP but no activation function in the output node? $\endgroup$
    – Dave
    Jun 18, 2021 at 17:06

1 Answer 1

1
$\begingroup$

Yes you are mostly correct. A feedforward neural network with a single layer and a sigmoid activation is a logistic regression which belongs to GLM type of models. Your second statement is unclear (weights interact with outputs) so I will try to break this down below:

Non-linear transformations (e.g. polynomial regression, logistic unit etc.) is often misread for non-linearity in model parameters (non-linear models).

As an example let's look at a feedforward neural network architecture. For $f(x)$ activation function and $w,b$ weights and biases, the output of a neuron from the first layer of a feed forward network would look like:

$$a_{11}=f(w_{11}x_1 + w_{12}x_2 + .. +b_1)$$

whereas the output of a neuron from the second layer neural of that neural network would look like:

$a_{21}=f(w_{21}a_{11}+ w_{22}a_{22} + .. + b_2)$, and given $a_1$ above $ \Rightarrow a_2=f(w_2w_1x+w_2b1+b2)$

$$a_{21}=w_{21}w_{11}x_1 + w_{21}w_{12}x_2+ .. + b_2$$

The multiplication between parameters (here $w_1w_2$) is what makes a model non-linear. In order to acquire that you need:

  • Either multiple layers OR
  • A non-linearity from the activation function e.g. if $f(x)=x^2$ even the output from the first layer of the neural network would be $a_{11}=w_{11}^2x^2 + b_2^2+w_{11}b_1x$ that qualifies as parameter multiplication in the $w_{11}^2$ factor and a non-linear model of a non-linear relationship with additive errors.

Hope it helps.

Sauces:

$\endgroup$
2
  • 1
    $\begingroup$ "Single-layer" is confusing. Do you mean a network with one hidden layer or zero? // Even a network with no hidden layers does not have to be a logistic regression. It could be a linear model or any other kind of generalized linear model (or perhaps even a weirder activation function than GLMs use as link functions). $\endgroup$
    – Dave
    Jun 21, 2021 at 13:15
  • $\begingroup$ Single layer neural network, single layer perceptron are widely used in the literature afaik, e.g.en.wikipedia.org/wiki/…. On your second point you are right, a sigmoid activation needs to be in place so that it is a logistic regression - i have amended this mistake now. $\endgroup$
    – hH1sG0n3
    Jun 21, 2021 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.