Why is the input to an activation function a linear combination of the input features?

Question

I'm new to deep learning, and there is something about the calculations I do not understand: For a standard neural network, why is it that only the activation function is not linear, but the input to the activation function is a linear combination of each of the $x_i$'s? For example, with the sigmoid function, it would look like: $$ \frac{1}{1+ e^{-(w_0x_0 + w_1x_1 + b)}} $$ where $w_i$ are the weights and $x_i$ represents the input to that layer.

For example, why is it that we don't have something like this: $$ \frac{1}{1+ e^{-(w_0x_0^2 + w_1\sqrt{x_1} + b)}} $$

Is it because it would be redundant if we had enough layers? Or is it because a priori, you wouldn't know what the best function is?

Answer 1

The main reason is that a linear combination of the input followed by a non-linearity stacked on top of eachother is a universal function approximator. Which means that no matter how complicated the true underlying function is, a neural network can approximate it to an arbitrarily small error.

There's also the efficiency factor since a linear combination of $n$ inputs each having $m$ dimensions can be represented using a single matrix multiplication $h=X \times W$ where $X$ is an $n \times m$ matrix (where each row is an example and each column is a feature of that example) and $W$ is an $ m \times d $ weight matrix. And computers are VERY efficient at doing matrix multiplications. Thus, the more you build your model to use matrix multiplications the better.

Answer 2

You can build that, for sure. I suppose it turns out that many layers of linear transformation with nonlinear activation work really well for common use cases, like responding to the intensity of pixels. Trying a ton of polynomial variations on the input doesn't work out well. But it could in some case. This is more a question of engineering the inputs to the neural net that make sense more than what you would want to do inside the network.