# Forward propagation in complicated neural networks

In a typical neural network you have a certain number of neurons in each layer, all neurons have incoming connections from each neuron in the previous layer (and visa-versa). This I have no problem figuring out how to propagate.

However when you have a network that doesn't have the neurons in layers (I'm looking into the NEAT algorithm); where neurons are just connected based on genetic algorithm (random creation - then testing of fitness) there isn't a simple way of figuring how to forward propagate, for example;

1 input neuron - connected to neurons 2 and 3
Neuron 2 is also connected to neuron 3
Neuron 3 is connected to the output.


Now in this case, if I just go through numerically then everything works out just fine. However with this subtle change;

1 input neuron - connected to neurons 2 and 3
Neuron **3** is also connected to neuron **2**
Neuron **2** is connected to the output.


The going through; 1, 2, then 3 would mean neuron 2 doesn't include the input from neuron 3. I can see that there's a brute force way through this, but is there a better, more efficient way?

• Note some architectures will take current output from a neuron as input to neurons connected to it, when "current" can mean "whatever has already been calculated in order" - potentially from a previous input or a learned start value. RNNs do this, and they can be used in control systems similar to NEAT, but I'm not at all sure what NEAT does. So if you are only interested in NEAT's approach, you should clarify because there is likely more than one way to resolve the issue. – Neil Slater Feb 9 '16 at 12:50

You can do topological sorting. For example, here is some pythonic pseudocode

def order(current_neuron, level):
current_neuron.level = max(level, current_neuron.level)
for child in sorted(current_neuron.children, key=children level, reverse=True):
order(child, level + 1)

for input_neuron in input_neurons:
order(input_neuron, level=0)


(Take this pseudo code with a grain of salt. This is just what came directly to my mind; it is not tested.)

Then you can calculate the activations in order of node.level.