Why there is only one type of artificial neuron?

Question

I find it strange that so many deep learning tricks and improvements have been invented in the past decade but I never heard about someone trying out different models of the artificial neuron other than the one based on perceptron:

y = w*x + b

I haven't been able to find much info on this online which is surprising. I am not a machine learning expert but from the little I know, it would make sense to me to at least experiment with other options. A trivial example, what would happen if there was a layer in a network consisting of neurons where

y = w*x^2 + b

Maybe there is an obvious answer to why my suggestion above isn't a good idea but I would prefer an answer that explains why generally this is not being looked into (at least as far as i know).

Answer 1

The example you cited (using x^2 instead of x) is the idea more popular outside deep learning community, called feature engineering. The trend in neural network modeling is instead to,

1. Play with weights (w) and fine tune them.
2. Not change the input vector (x) but feed it to the network directly.
3. If a single layer neural network is not good enough, add more layers.
4. Introduce non-linearities using activation functions.
5. And in general, not hand-roll features (like x^2) but let neural networks discover such features.

Answer 2

I think that the "neurons" analogy is not very helpful to understand what is going on with artificial neural networks.

Neural networks are not comprised by "neurons", but by differentiable operations. These operations are arbitrary, e.g. convolutions, indexing (in embeddings), pooling, etc.

What you proposed is a perfectly valid building block of a hypothetical neural network.

The classical "neuron" analogy is used for explaining the multilayer perceptron (MLP), which is a mere sequence of fully connected layers with non-linear activations in between. As soon as you depart from the simple MLP, applying the neuron analogy becomes more cumbersome.

Answer 3

There can be many answers for this question but probably cause it is just an unnecessary complexity.

You can achieve same result (x2) with the current architecture (i.e. using multiplication layer; not to mention just squaring your input features). Why would you use something more specific and not something more general?

Beside that why would your final result benefit from it? You can introduce non-linearities in many other ways. That is the purpose of the activation function.