After thinking about a question comparing Bayesian Network to Neural Network, I am now wondering if they may not be one and the same thing!

At this point, my maths (unused for over 20 years fails me!). But....

  • If infinite random test data was used
  • The output of the Bayesian Network was found for each data item
  • And the Neural Network was then trained on it

Would the Neural Network not tent towards being the same as the Bayesian Network?


We don't have to do anything crazy to prove this, we can simply leverage the UAT.

By the universal approximation theorem, yes, you could concieve of a neural network that precisely matched your Bayesian network.

  • $\begingroup$ can it be done the other way round so a Bayes network matches any given neural network? $\endgroup$ – Ian Ringrose Jan 19 '16 at 19:20
  • $\begingroup$ Note that this says nothing about the size of the neural network required; it is not a practical result. $\endgroup$ – Emre Jan 20 '16 at 3:53

I would say Yes.

As Jacob neatly put, the Universal Approximation algo. can be leveraged for proving the fact.

In fact, as neural networks are highly flexible, owing to their portable structure definitions and a huge set of activation functions one can choose from, I don't see why a Bayesian network cannot be mimicked by a neural network. But, from my experience of both research and coding deep belief networks, I'd advise you not to try this at home. Cause, you would often end up with painfully complex networks. :D

But, I am skeptical about the vice-versa process. (owing to my limited knowledge of Bayesian networks)


As pointed out by Jacob Panikulam, Universal Approximation Theorem gives you the answer. So...

Yes. But please do note that anyway this is valid for GLMs, SVMs... So...

Would the Neural Network not tent towards being the same as the Bayesian Network?

Yes, it does, as does with any other model.

Note that this does not happen the other way around, Bayesian Networks cannot model certain types of dependencies, e.g. circular dependencies. Also, all Bayesian Network structure learning algorithms assume faithfulness, so they perform pretty badly with certain types of independent and conditionally dependent variables: see Is there a Probabilistic Graphical Model for this Situation?

Caution Note: Please do not mistake this with the fact that you should only learn about and use neural networks. Introduction to Statistical Learning thoroughly explains why.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.