# Can we think of neurons as maps between matrices?

Usually when we think about neurons, we imagine that they enact some kind of map between real numbers. For example, a neuron might take in real numbers $$x_{i}$$ and weight them with parameters $$W_{ij}$$, so that the neuron $$j$$ in a subsequent layer receives an input $$\sum_{i}W_{ij}x_{i}$$ and outputs $$f(\sum_{i}W_{ij}x_{i}+b_{j})$$, where $$b_{j}$$ is a bias and $$f$$ is the activation function. However, one can instead imagine a neuron that maps matrices to matrices by simply promoting the outputs $$x_{i}$$ and biases $$b_{j}$$ to matrices, so that they carry an additional index structure, e.g., $$x_{i}\rightarrow x_{i}^{\alpha\beta}$$. In this convention, lower indices would then indicate the locations of objects in the network and upper indices would indicate matrix elements, so that $$x_{i}^{\alpha\beta}$$ denotes a matrix output from neuron $$i$$ with matrix elements indexed by $$\alpha$$ and $$\beta$$.

This kind of structure then tells us that an input to an arbitrary neuron, i.e. $$\sum_{i}W_{ij}x_{i}^{\alpha\beta}$$, is just a linear combination of the matrices output by the previous layer of the network (note that the weights are still real numbers). Further, the activation function $$f$$ would now be a matrix function, by which I mean the output of a neuron would be defined by the Taylor series of that function evaluated on the input matrix.

Could such a scheme work, by which I mean could such a network be practically trainable? It seems to me that the answer should be yes if the activation functions in the network are well behaved. A simple example would be something like the matrix exponential function: suppose I want to compute, for a matrix $$A$$,

$$e^{A} = \sum_{n=0}^{\infty}\frac{1}{n!}A^{n}$$

Further, $$A$$ can be written as some linear combination of other matrices $$A_{i}$$: $$A = \sum_{i}a_{i}A_{i}$$, with the $$a_{i}$$ real numbers. If I fix $$e^{A}$$, shouldn't it be possible to train a network by adjusting the $$a_{i}$$ until a best fit is found?

This seems like a plausible idea but I'm not aware of neurons being used in this way. Has this been discussed before in the literature?

EDIT: I have been asked to explain the purpose of decomposing $$A$$ into the basis given by the $$A_{i}$$. The order of operations I have in mind is this: suppose you are given a matrix $$B = f(A)$$, with $$f$$ a nonlinear function. I can try to find a matrix $$A'$$ which generates a $$B'$$ that closely approximates $$B$$. But what if I want $$A'$$ to have a particular form? In other words, make this into a constrained optimization problem. One way to do this is to pick a basis, which I called $$A_{i}$$ previously, and truncate it - this forces you to find the optimal $$A'$$ in the restricted basis. (This has lots of possible applications in physics that I can discuss but for now I'll just claim this is an interesting thing to do.)

• I don't get the point in decomposing $A$ into a linear combination of $A_i$. When training the network what you adjust are the weights $W_{ij}$. Could you explain that further? – Javi Feb 2 at 15:08
• @Javi Certainly. I'm still proposing that we adjust only the weights/biases. But in many situations in linear algebra, one can decompose an operator into a basis. Given a basis $A_{i}$ it's useful to have a procedure for determining the expansion coefficients $a_{i}$. This can often be done analytically by choosing a "nice" basis, but what if I fix $f(A)$ instead of $A$ so that the decomposition you want is related in a nonlinear way to the given matrix? I'll write more details about this in the question as well. – miggle Feb 2 at 17:44