# Why neural networks models do not allow for multiplication of inputs?

In a neural network, each neuron value is multiplied by the weight of the connection. Then, each neuron's input is the sum of all those values, on which we apply the activation function (sigmoid, relu, tanh, etc.).

Why can't the input be a vector of all those values, so that I can decide to multiply them if I want, or apply another function from the vector space to the reals, and then apply the activation function on that real value? Any implementation I have seen does not allow this, and if I wanted to do it, it seem I would have to implement my whole neural network architecture by myself without the possibility of using Theano, Keras or other libraries. Is there a reason that escapes me of why this would not work? And if there isn't, is there any library out there I could use that already does this without having to create it from scratch?

Using a low-level library, such as Theano or TensorFlow, it is likely that you can construct new schemes for reducing tensors (maybe via some learnable weight vector etc). In TensorFlow, you also get automated gradient calculations, so you should be able to still define a cost function and use existing optimisers, without needing to analyse the new design yourself or re-write back propagation formulae.

The universal approximation theorem essentially states that in order to have a network that could learn a specific function, you don't need anything more than one hidden layer using the standard matrix multiplying plus non-linear activation function. So, invention and use of new architectures needs some reason. The reason is usually that these additions improve the speed or scope of what can be learned, they generalise better from training data, or they model something from a problem domain really well.

No doubt there have been explorations of all sorts of variations to the standard NN model over time. The ones that have become popular have all proven themselves on some task or other, and often have associated papers demonstrating their usefulness.

For example Convolutional Neural Networks have proven themselves good at image-based tasks. The design of a 2-dimensional CNN layer has a logical match to how pixels in an image relate to each other locally - defining edges, textures etc, so the architecture in the model nicely matches to some problem domains.

If you can find a good match from your vector multiplication model to a specific problem domain, then that's an indication that it may be worth implementing in order to test that theory. If you just want to explore other NN structures to see whether they work, you can still do so, but without a specific target you will be searching for a problem to solve (unless by chance you stumble upon something generally useful that has been overlooked before).

To address the question in the title:

Why neural networks models do not allow for multiplication of inputs?

Some low-level ones do (e.g. TensorFlow). However, this is not an architecture that has proven itself useful, so authors of higher-level libraries (e.g. Keras) have no reason to implement or support it. There is the possibility that this situation is an oversight, and the idea is generally useful. In which case, once someone can show this you will find it would get support across actively-developed libraries pretty quickly since it seems straightforward to implement.

This is the standard definition of matrix multiplication. Summing those values I mean. You don't have to make it work like that, you can in fact multiply them if you like. Simply do a series of scalar multiplications instead:

# Input vector
X = tf.placeholder("float32", (-1, 30)) # so a 1x30 input vector

# "Layer 1"
w_l1 = []
for n in range(50):
w_l1.append(tf.Variable(tf.random_normal((1,))))

l1 = []
for n in range(50):
l1.append(X * w_l1[n])

l1_out = l1[0]
for n in range(1, 50):
l1_out *= l1[n]


Is this close to what you mean ?

More generally, I think most of what you want to achieve can be done with the reduce_prod method in tensorflow, and theano has an analogous function. I think it isn't done in practice because it doesn't form a ring over Tensors.

This paper states (verbatim):

... multiplication and division can be computed by depth-3 ANNs

• That's not the same thing. That paper deals with the case where the values $x,y$ to be multiplied are represented in binary. The question is asking about the case where $x,y$ are each present on one wire (not in binary, but directly as is). – D.W. Jun 9 '18 at 16:04

Why can't the input be a vector of all those values, so that I can decide to multiply them if I want, or apply another function from the vector space to the reals, and then apply the activation function on that real value?

You CAN implement that in Keras, using a custom Lambda layer, see this.

So assume that you have a feedforward neural network with 2 layers, which are fully connected. In Keras this means that you have 2 Dense layers sequentially. Each node of each layer performs a sum of its inputs and thereafter applies the activation function. If you wish to multiply the outputs of the first layer and then feed their products to the next Dense layer, you just need to insert a Lambda layer in between. The new Lambda layer shall multiply -or do whatever you want with- the outputs of the first layer and then feed the result in the other Dense layer.

Any implementation I have seen does not allow this, and if I wanted to do it, it seem I would have to implement my whole neural network architecture by myself without the possibility of using Theano, Keras or other libraries

Essentially, you just need to write only your custom Lambda layer and model.add() it in between any two layers of interest :)

The "universal approximation" of a standard neural network is not universal. The small letters tell that the target function must be bounded, and that you may need a really wide hidden layer to get a big value range, when the output of each unit is bounded to e.g. (-1,1). Then comes the issue of efficient learning, because ability to represent does not mean ability to learn efficiently. Exploring other types of neurons can be useful.

A concrete application of multiplicative neurons could be to have an initial layer that generates an arbitrary number of compounded features from the inputs, generalizing beyond a fixed set of polynomial terms with non-negative integer exponents below a given value. The next layers could consist of standard additive neurons, giving linear combinations of the features.

Multiplicative neurons have been explored. Also neurons that interpolate (at least approximately) between addition and multiplication:

The last one, NALU from Deepmind, apparently is a great success, and exists in community implementations for several NN frameworks. It can be plugged in for a linear layer, although costs more to compute, and has more parameters to train.

The logarithm of a product is the sum of logarithms of the factors. So one way to make a multiplicative neuron layer for positive inputs, is to apply a logarithm to the inputs, then run them through a linear layer, then exponentiate the output. The weights will correspond to exponents.