0
$\begingroup$

I'm new to deep learning, and there is something about the calculations I do not understand: For a standard neural network, why is it that only the activation function is not linear, but the input to the activation function is a linear combination of each of the $x_i$'s? For example, with the sigmoid function, it would look like: $$ \frac{1}{1+ e^{-(w_0x_0 + w_1x_1 + b)}} $$ where $w_i$ are the weights and $x_i$ represents the input to that layer.

For example, why is it that we don't have something like this: $$ \frac{1}{1+ e^{-(w_0x_0^2 + w_1\sqrt{x_1} + b)}} $$

Is it because it would be redundant if we had enough layers? Or is it because a priori, you wouldn't know what the best function is?

$\endgroup$
2
$\begingroup$

The main reason is that a linear combination of the input followed by a non-linearity stacked on top of eachother is a universal function approximator. Which means that no matter how complicated the true underlying function is, a neural network can approximate it to an arbitrarily small error.

There's also the efficiency factor since a linear combination of $n$ inputs each having $m$ dimensions can be represented using a single matrix multiplication $h=X \times W$ where $X$ is an $n \times m$ matrix (where each row is an example and each column is a feature of that example) and $W$ is an $ m \times d $ weight matrix. And computers are VERY efficient at doing matrix multiplications. Thus, the more you build your model to use matrix multiplications the better.

$\endgroup$
0
$\begingroup$

You can build that, for sure. I suppose it turns out that many layers of linear transformation with nonlinear activation work really well for common use cases, like responding to the intensity of pixels. Trying a ton of polynomial variations on the input doesn't work out well. But it could in some case. This is more a question of engineering the inputs to the neural net that make sense more than what you would want to do inside the network.

$\endgroup$
1
  • $\begingroup$ Is this just a result of experience, or is there some theory behind why linear transformation + non linear activation works well? $\endgroup$
    – Snowball
    May 1 '20 at 7:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.