1
$\begingroup$

First of all Thank you for taking your time to answer my questions.

Disclaimer: I am no where near expert. I am just an undergrad student who is planning to get PhD in Deep Learning.

I have been thinking a lot about new ways of creating network that represent a real brain that could possible be used in place of traditional Deep learning architecture.

At first, I think it is too ambitious and out of my reach, but the more I read about it the more excitement I get from the possibility, so I just created this new account to specifically ask this question.

At first, I notice that traditional Deep Learning architecture does not quite resemble neural firing patterns. enter image description here

and this enter image description here

You can see from pictures of brain activity graph above that neurons within the same area of a brain tends to cluster together.

NOW compare this to traditional Deep learning (DL) architecture below. You know that different layers works differently (in general, you get the idea) for example, output layers use sigmold as activation function while hidden layers uses Relu. Then, it is fair to say that neuron within the same layers works similarly. Using the same argument, it is also fair to argue that DL networks do not quite resemble the actual neural activity network within the brain itself which is what it is said to be inspired by.
enter image description here

Then after some more reading about how emotion, memory, and brain works in general. I realized that there is another false in the design to resemble real neural network. First is degeneracy which says, in not so accurate term, that, assuming all input are the same, there could always be more than 1 cascade neural activity which lead to the same output.

From the above, we can see that DL is not accurate. First, it only have one fix path to construct output given the exact same input. (we are talking about constructing output because during training, dropout attempts to mimic degeneracy). Second, DL network does not have state. By this, I mean DL is just non-linear transformation function it has no any specific patterns ingrain within each neuron. (sure, some other archtecture like LSTM has cell state. Still, it doesn't satisfy all other requirements I am looking for)

Then, I came across something call Cellular Automaton which shows that simple, plain patterns can create 1. uniform 2.repetition 3.random 4.complicated patterns. One of the property of Cellular Automaton that interests me is that it could allow for larger search space in comparison to DL.

So, by using this, we can train neurons for certain tasks while preserving its ability to generalize.

Now, after a little more thinking, I realized that it is also possible that neurons can contains the exact same random patterns and order in which firing cascade occurs could create extremely complicated patterns. This aligns with my previous believe that each neural should obtain random patterns (more random patterns allows for more possibility of constructing more complicated patterns) which is heavily influenced by Carlos E. Perez.

This is one of the video from Steven Wolfram whose believe and approach does not quite follow traditional believe of path of AI.

MIT AGI: Computational Universe (Stephen Wolfram)

here is another quick video that discuss the topological shape of connected neural graph.

Your Mind Is Eight-Dimensional - Your Brain as Math Part 3 | Infinite Series

I am still in process on learning Algebraic Topology, so words i will be using will not be as accurate, but please hear me out.

The second video basically says that when the brain ( in this case, stimulated brain from Blue Brain project) is stimulated, Directed Simplices of the connected network is shown to not be formed by chanced and below is its properties as time increase measured by 1. number of edges 2. Betti numbers 3.Euler Characteristic.

enter image description here

Sorry for long post. I am really looking for specific types of answer.

So here is my question,

What kinds of math/equations/functions/rules/networks (whatever that is helpful will do) do you think I should know before attempting to create a graph which could obtains topological shape measure by 1. number of edges 2. Betti numbers 3.Euler Characteristic at each time step while allows itself to learn from feedback of its mistake?(algorithms such as back propagation)

I have absolutely no idea what to study or where to seek advice because I am not familiar with math major and don't know any experts who could give me advice.

$\endgroup$