Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
After thinking about a question comparing Bayesian Network to Neural Network, I am now wondering if they may not be one and the same thing!
At this point, my maths (unused for over 20 years fails me!). But....
Would the Neural Network not tent towards being the same as the Bayesian Network?
We don't have to do anything crazy to prove this, we can simply leverage the UAT.
By the universal approximation theorem, yes, you could concieve of a neural network that precisely matched your Bayesian network.
I would say Yes.
As Jacob neatly put, the Universal Approximation algo. can be leveraged for proving the fact.
In fact, as neural networks are highly flexible, owing to their portable structure definitions and a huge set of activation functions one can choose from, I don't see why a Bayesian network cannot be mimicked by a neural network. But, from my experience of both research and coding deep belief networks, I'd advise you not to try this at home. Cause, you would often end up with painfully complex networks. :D
But, I am skeptical about the vice-versa process. (owing to my limited knowledge of Bayesian networks)
As pointed out by Jacob Panikulam, Universal Approximation Theorem gives you the answer. So...
Yes. But please do note that anyway this is valid for GLMs, SVMs... So...
Would the Neural Network not tent towards being the same as the Bayesian Network?
Yes, it does, as does with any other model.
Note that this does not happen the other way around, Bayesian Networks cannot model certain types of dependencies, e.g. circular dependencies. Also, all Bayesian Network structure learning algorithms assume faithfulness, so they perform pretty badly with certain types of independent and conditionally dependent variables: see Is there a Probabilistic Graphical Model for this Situation?
Caution Note: Please do not mistake this with the fact that you should only learn about and use neural networks. Introduction to Statistical Learning thoroughly explains why.
