Ideal aggregation function for Partially Connected Neural Network (PCNN)

Question

I am building a Python library that creates Partially Connected Neural Networks based on input and output data (X,Y). The basic gist is that the network graph is arbitrarily updated with nodes and edges in the hidden layers.

Example Data:

X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=np.float)
Y = np.array([[0],[1],[1],[0]], dtype=np.float)

Example Graph:

I am currently using the sum product aggregation function to calculate each layer's values:

$$ \sum_{i=0}^n w_{ij}x_i $$

The synapse weights are denoted by two subscripts $ij$. Subscript $i$ represents the previous neuron and subscript $j$ represents the current neuron under consideration. The sum product for current neuron $j$ is computed as:

$$ s_j = w_{0j}x_0 + w_{1j}x_1 + ... + w_{nj}x_n = \sum_{i=0}^n w_{ij}x_i $$

Is sum product an ideal aggregation function for PCNNs? Dot product won't work as the tensor sizes are almost always incompatible.

Update: As Bence Mélykúti suggested, my solution was to use a weight vector $w$ that is padded with zeros and reshaped for dot product. Here is how I accomplished it in my Python library:

from functools import reduce
import numpy as np
import math

# An example of some input/sensor data
X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=np.float16)
# From the graph, two inbound synapses in layer 1, here randomly generated
W = np.random.random((2,))

# Enumerate the element count in each array
xElements = reduce(lambda x, y: x*y, list(X.shape))
wElements = reduce(lambda x, y: x*y, list(W.shape))

# If previous layer has more elements than the weights in the current layer
if xElements > wElements:
    # Create a list of zeros of shape X.shape
    A = np.zeros(X.shape).flatten().tolist()
else:
    # Otherwise, create a list with the larger number of elements in W
    A = np.zeros(W.shape).flatten().tolist()

# If the list is odd, add one element to ensure it is even
# Done so the array can be reshaped to fit the data and dot product
if len(A) % 2 != 0:
    A.extend([np.float16(0.0)])

# Fill the list with the elements from W
for ix,ele in enumerate(W):
    A[ix] = ele
A = np.array(A)

# Convert list to an array and reshape to be compatible with dot product
A = A.reshape(X.shape[-1],(math.floor(A.shape[0]/2)))

result = np.dot(X,A)

The result of the dot product looks like:

print(result)
array([[0.        , 0.        , 0.        , 0.        ],
       [0.        , 0.        , 0.        , 0.        ],
       [0.45510922, 0.79058734, 0.        , 0.        ],
       [0.45510922, 0.79058734, 0.        , 0.        ]])

Thanks Bence!

Answer 1

What you call sum product is the same as a dot product of two vectors of equal length, $(w_{0j},w_{1j},\dots,w_{nj})$ and $(x_0,x_1,\dots,x_n)$.

Based on your diagram, your difficulty is that you want to be able to define connections between neurons that are not in consecutive layers (in other words, the connections might skip layers). I assume that there are no directed edges (synapses) that would go in the opposite direction (from layer $n$ to layer $m$ for some $m<n$). In this setting, you have to forward propagate information to layer $n$ from input $X$ and layers $1,2,\ldots,n-1$. I don't know when and how often you want to 'update' your network topology. Do you want to change it during training?! (Otherwise you'd fix a partially connected NN and you wouldn't write update in your question.)

For forward propagation in a fixed network topology, I would associate to neuron $j$ a vector of weights $(w_{ij})_{i\in I_j}$, where $I_j$ is the set of neurons that bring information to $j$ (those $i$ where there exists a directed edge from $i$ to $j$). Then you could either cherry-pick the neurons $i$ from the set of all neurons (by maintaining a list of indexes to find them in the vector of all neurons) and put them into a vector $x$, which is now as long as $(w_{ij})_{i\in I_j}$, and you can use dot product on the two vectors. (With numpy for Python, numpy.dot would work.) You would need to do this separately for all $j$ in the given layer. As these are receiving information from differing numbers of inputs and neurons, they indeed cannot be arranged into a matrix/tensor, not even receiving neurons of the same layer.

Or you could force all potential preceding neurons into one vector. Neuron $j$ of layer $n$ can only receive information from neurons $x=(<$input nodes$>$, $<$neurons of layer $1>$, $\ldots$, $<$neurons of layer $n-1>)$. This is a vector, and you can define a vector $w$ of equal length for $j$, which would indeed have many zeros, and $x$ and $w$ can be multiplied. For all neurons of layer $n$, you could stack these $w$ as row vectors on top of each other into a matrix/tensor of size (the number of neurons in layer $n$) times (the size of input $+$ the number of neurons in layer $1$ $+\ldots+$ the number of neurons in layer $n-1$) and use np.dot between this matrix and the $x$.

How to do training (backpropagation) with such topologies is something you have to consider a bit. I suspect it will work without much adjustment compared to a fully connected neural network without layer skipping. In the second option, you would only need to zero out the weights which must be kept zero after each backpropagation update. This indeed carries overhead if you're not careful. But if you maintained a binary ($0$ or $1$) 'mask' matrix of which weights are live (which correspond to existing connections), then you would only ever update these weights in the backpropagation.

Additionally, you may want to introduce a bias term $w_{-1j}$ (also known as intercept term, a constant offset) as if $x$ had an extra dimension that is kept constant $1$. After the aggregation with the sum product, you should also pass the result through a non-linear activation function before entering the value into the neuron in the next layer.