I'm new to deep learning, and there is something about the calculations I do not understand: For a standard neural network, why is it that only the activation function is not linear, but the input to the activation function is a linear combination of each of the $x_i$'s? For example, with the sigmoid function, it would look like: $$ \frac{1}{1+ e^{-(w_0x_0 + w_1x_1 + b)}} $$ where $w_i$ are the weights and $x_i$ represents the input to that layer.
For example, why is it that we don't have something like this: $$ \frac{1}{1+ e^{-(w_0x_0^2 + w_1\sqrt{x_1} + b)}} $$
Is it because it would be redundant if we had enough layers? Or is it because a priori, you wouldn't know what the best function is?