Usually when we think about neurons, we imagine that they enact some kind of map between real numbers. For example, a neuron might take in real numbers $x_{i}$ and weight them with parameters $W_{ij}$, so that the neuron $j$ in a subsequent layer receives an input $\sum_{i}W_{ij}x_{i}$ and outputs $f(\sum_{i}W_{ij}x_{i}+b_{j})$, where $b_{j}$ is a bias and $f$ is the activation function. However, one can instead imagine a neuron that maps matrices to matrices by simply promoting the outputs $x_{i}$ and biases $b_{j}$ to matrices, so that they carry an additional index structure, e.g., $x_{i}\rightarrow x_{i}^{\alpha\beta}$. In this convention, lower indices would then indicate the locations of objects in the network and upper indices would indicate matrix elements, so that $x_{i}^{\alpha\beta}$ denotes a matrix output from neuron $i$ with matrix elements indexed by $\alpha$ and $\beta$.

This kind of structure then tells us that an input to an arbitrary neuron, i.e. $\sum_{i}W_{ij}x_{i}^{\alpha\beta}$, is just a linear combination of the matrices output by the previous layer of the network (note that the weights are still real numbers). Further, the activation function $f$ would now be a matrix function, by which I mean the output of a neuron would be defined by the Taylor series of that function evaluated on the input matrix.

Could such a scheme work, by which I mean could such a network be practically trainable? It seems to me that the answer should be yes if the activation functions in the network are well behaved. A simple example would be something like the matrix exponential function: suppose I want to compute, for a matrix $A$,

$$e^{A} = \sum_{n=0}^{\infty}\frac{1}{n!}A^{n}$$

Further, $A$ can be written as some linear combination of other matrices $A_{i}$: $A = \sum_{i}a_{i}A_{i}$, with the $a_{i}$ real numbers. If I fix $e^{A}$, shouldn't it be possible to train a network by adjusting the $a_{i}$ until a best fit is found?

This seems like a plausible idea but I'm not aware of neurons being used in this way. Has this been discussed before in the literature?

EDIT: I have been asked to explain the purpose of decomposing $A$ into the basis given by the $A_{i}$. The order of operations I have in mind is this: suppose you are given a matrix $B = f(A)$, with $f$ a nonlinear function. I can try to find a matrix $A'$ which generates a $B'$ that closely approximates $B$. But what if I want $A'$ to have a particular form? In other words, make this into a constrained optimization problem. One way to do this is to pick a basis, which I called $A_{i}$ previously, and truncate it - this forces you to find the optimal $A'$ in the restricted basis. (This has lots of possible applications in physics that I can discuss but for now I'll just claim this is an interesting thing to do.)

  • $\begingroup$ I don't get the point in decomposing $A $ into a linear combination of $A_i $. When training the network what you adjust are the weights $W_{ij} $. Could you explain that further? $\endgroup$ – Javi Feb 2 at 15:08
  • $\begingroup$ @Javi Certainly. I'm still proposing that we adjust only the weights/biases. But in many situations in linear algebra, one can decompose an operator into a basis. Given a basis $A_{i}$ it's useful to have a procedure for determining the expansion coefficients $a_{i}$. This can often be done analytically by choosing a "nice" basis, but what if I fix $f(A)$ instead of $A$ so that the decomposition you want is related in a nonlinear way to the given matrix? I'll write more details about this in the question as well. $\endgroup$ – miggle Feb 2 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.