6
$\begingroup$

In the section about batch normalization of Deep Learning book by Ian Goodfellow (chapter link) there is the follwing text:

As example, suppose we have a deep neural network that has only one unit per layerand does not use an activation function at each hidden layer: $y=x w_1 w_2 w_3 \ldots w_l$. Here, $w_i$ provides the weight used by layer $i$. The output of layer $i$ is $h_i=h_{i−1} wi$. The output $y$ is a linear function of the input $x$, but a nonlinear function of the weights $w_i$.

Why y is nonlinear with respect to w_i?

$\endgroup$
  • 1
    $\begingroup$ Sorry, I didn't read well. One unit per layer means one neuron. Then w_i are scalar, not necessarily x and y. Then I don't really understand this statement. I delete my answer. $\endgroup$ – debzsud Feb 18 '17 at 17:27
2
$\begingroup$

I think what the statement meant was when given weights $w_1,...,w_n$ are fixed, output is linear proportional to $x$, but as it mentions

nonlinear function of the weights w_i

Given a set of weights (more than one being varied), they do not linearly add to produce an output like

$y=w_1x_1+...+w_nx_n$

but rather non-linearly like

$y=w_1w_2..w_n*x$ (each $w_i$ is a dimension in the hyperspace)

And I think it would become more clearer from the statement "Output is linear to any weight $w_i$ but non-linear to weights $w_i$".

$\endgroup$
  • $\begingroup$ I'm not sure i understand the answer. what does the ∗ operator represents I we have one unit per layer, each w_i is a scalar, so how can it be not linear? $\endgroup$ – amit Feb 23 '17 at 13:28
  • $\begingroup$ @amit * is just a multiplication operator. In the case which you did not understand, $w_i$ is no scalar but a variable. The statement is saying that if $w_i$'s are variable and lets consider them as axis of nD space, changing some of them would not produce a linear output since they are getting multiplied. But keeping all weights constant except one weight $w_i$, so any changes in that weight produce a linear output, in this case. $\endgroup$ – Kiritee Gak Feb 23 '17 at 13:34
  • $\begingroup$ I still fail to understand. What do you mean by considering them as axis of nD space? Each w_i is a scalar variable representing one dimension in some W vector? How multiplying them make sense in that context? $\endgroup$ – amit Feb 23 '17 at 17:17
  • $\begingroup$ Okay, my bad. They are scalars but are variables. Forget the axis too. Simply putting, Is $z =k*x*y$ linear? if k is a constant and x,y are variables? No. Taking this case as an equivalence of the network, $x,y$ are weights of the network with $k$ being the $x$ in the last equation of my answer. So $x,y$ are in a non-linear equation. Are you able to compare this $z=k*x*y$ with $y=x*w_1*w_2..*w_n$ and see what are the variables(weights) and why the equation is non-linear with respect to weights $w_i$. If you are still unclear, we can extend it in chat. $\endgroup$ – Kiritee Gak Feb 23 '17 at 18:19
1
$\begingroup$

suppose $w_1 = w_2 =... w_n = w$ then $y = w^n \times x $. In this sense is $y$ a linear function of $x$ and a non-linear function of $w$.

$\endgroup$
  • $\begingroup$ Can you explain how this differs from the accepted answer? $\endgroup$ – Stephen Rauch Dec 24 '17 at 19:39
  • $\begingroup$ it does not differ that much. It's just a different perspective. $\endgroup$ – P.Joseph Dec 25 '17 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.