In the section about batch normalization of Deep Learning book by Ian Goodfellow (chapter link) there is the follwing text:

As example, suppose we have a deep neural network that has only one unit per layerand does not use an activation function at each hidden layer: $y=x w_1 w_2 w_3 \ldots w_l$. Here, $w_i$ provides the weight used by layer $i$. The output of layer $i$ is $h_i=h_{i−1} wi$. The output $y$ is a linear function of the input $x$, but a nonlinear function of the weights $w_i$.

Why y is nonlinear with respect to w_i?

  • 1
    $\begingroup$ Sorry, I didn't read well. One unit per layer means one neuron. Then w_i are scalar, not necessarily x and y. Then I don't really understand this statement. I delete my answer. $\endgroup$
    – Robin
    Feb 18, 2017 at 17:27

2 Answers 2


I think what the statement meant was when given weights $w_1,...,w_n$ are fixed, output is linear proportional to $x$, but as it mentions

nonlinear function of the weights w_i

Given a set of weights (more than one being varied), they do not linearly add to produce an output like


but rather non-linearly like

$y=w_1w_2..w_n*x$ (each $w_i$ is a dimension in the hyperspace)

And I think it would become more clearer from the statement "Output is linear to any weight $w_i$ but non-linear to weights $w_i$".

  • $\begingroup$ I'm not sure i understand the answer. what does the ∗ operator represents I we have one unit per layer, each w_i is a scalar, so how can it be not linear? $\endgroup$
    – amit
    Feb 23, 2017 at 13:28
  • $\begingroup$ @amit * is just a multiplication operator. In the case which you did not understand, $w_i$ is no scalar but a variable. The statement is saying that if $w_i$'s are variable and lets consider them as axis of nD space, changing some of them would not produce a linear output since they are getting multiplied. But keeping all weights constant except one weight $w_i$, so any changes in that weight produce a linear output, in this case. $\endgroup$ Feb 23, 2017 at 13:34
  • $\begingroup$ I still fail to understand. What do you mean by considering them as axis of nD space? Each w_i is a scalar variable representing one dimension in some W vector? How multiplying them make sense in that context? $\endgroup$
    – amit
    Feb 23, 2017 at 17:17
  • $\begingroup$ Okay, my bad. They are scalars but are variables. Forget the axis too. Simply putting, Is $z =k*x*y$ linear? if k is a constant and x,y are variables? No. Taking this case as an equivalence of the network, $x,y$ are weights of the network with $k$ being the $x$ in the last equation of my answer. So $x,y$ are in a non-linear equation. Are you able to compare this $z=k*x*y$ with $y=x*w_1*w_2..*w_n$ and see what are the variables(weights) and why the equation is non-linear with respect to weights $w_i$. If you are still unclear, we can extend it in chat. $\endgroup$ Feb 23, 2017 at 18:19

suppose $w_1 = w_2 =... w_n = w$ then $y = w^n \times x $. In this sense is $y$ a linear function of $x$ and a non-linear function of $w$.

  • $\begingroup$ Can you explain how this differs from the accepted answer? $\endgroup$
    – Stephen Rauch
    Dec 24, 2017 at 19:39
  • $\begingroup$ it does not differ that much. It's just a different perspective. $\endgroup$
    – P.Joseph
    Dec 25, 2017 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.