1
$\begingroup$

I find it strange that so many deep learning tricks and improvements have been invented in the past decade but I never heard about someone trying out different models of the artificial neuron other than the one based on perceptron:

y = w*x + b

I haven't been able to find much info on this online which is surprising. I am not a machine learning expert but from the little I know, it would make sense to me to at least experiment with other options. A trivial example, what would happen if there was a layer in a network consisting of neurons where

y = w*x^2 + b

Maybe there is an obvious answer to why my suggestion above isn't a good idea but I would prefer an answer that explains why generally this is not being looked into (at least as far as i know).

$\endgroup$
1
$\begingroup$

The example you cited (using x^2 instead of x) is the idea more popular outside deep learning community, called feature engineering. The trend in neural network modeling is instead to,

  1. Play with weights (w) and fine tune them.
  2. Not change the input vector (x) but feed it to the network directly.
  3. If a single layer neural network is not good enough, add more layers.
  4. Introduce non-linearities using activation functions.
  5. And in general, not hand-roll features (like x^2) but let neural networks discover such features.
$\endgroup$
1
  • $\begingroup$ Thanks for your answer! .. if we consider the formula of the artificial neuron, y = x*w + b as f(), so that y = f(x). Then when I suggested to use x^2 instead of x, that is feature engineering. You are right. But aren't there any other options for f()? $\endgroup$ – Jan Pisl Feb 5 at 10:47
1
$\begingroup$

I think that the "neurons" analogy is not very helpful to understand what is going on with artificial neural networks.

Neural networks are not comprised by "neurons", but by differentiable operations. These operations are arbitrary, e.g. convolutions, indexing (in embeddings), pooling, etc.

What you proposed is a perfectly valid building block of a hypothetical neural network.

The classical "neuron" analogy is used for explaining the multilayer perceptron (MLP), which is a mere sequence of fully connected layers with non-linear activations in between. As soon as you depart from the simple MLP, applying the neuron analogy becomes more cumbersome.

$\endgroup$
0
$\begingroup$

There can be many answers for this question but probably cause it is just an unnecessary complexity.

You can achieve same result (x2) with the current architecture (i.e. using multiplication layer; not to mention just squaring your input features). Why would you use something more specific and not something more general?

Beside that why would your final result benefit from it? You can introduce non-linearities in many other ways. That is the purpose of the activation function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.