60
$\begingroup$

Neural networks get top results in Computer Vision tasks (see MNIST, ILSVRC, Kaggle Galaxy Challenge). They seem to outperform every other approach in Computer Vision. But there are also other tasks:

I'm not too sure about ASR (automatic speech recognition) and machine translation, but I think I've also heard that (recurrent) neural networks (start to) outperform other approaches.

I am currently learning about Bayesian Networks and I wonder in which cases those models are usually applied. So my question is:

Is there any challenge / (Kaggle) competition, where the state of the art are Bayesian Networks or at least very similar models?

(Side note: I've also seen decision trees, 2, 3, 4, 5, 6, 7 win in several recent Kaggle challenges)

$\endgroup$
6
  • $\begingroup$ It's not a question of domain. It's a question of how much data you have, how good your priors are, and whether you want posteriors. $\endgroup$
    – Emre
    Jan 17, 2016 at 18:22
  • 1
    $\begingroup$ @Emre Which is a question of domain... (and, of course, of money when you have the possibility to not only use existing datasets but can also hire people to create / label new data). $\endgroup$ Jan 17, 2016 at 18:26
  • $\begingroup$ It would be a question of domain if there were some property of the data, some structure, that one algorithm took advantage of better than the other, but that is not what I am suggesting. $\endgroup$
    – Emre
    Jan 17, 2016 at 19:16
  • 2
    $\begingroup$ So the answer to your question is then, No. Right? Because all answers seem to point the advantages of Bayesian Networks over other predictive models, but I have not seen any Kaggle competition where they actually outperform other models. Can anyone provide one? Because all the reasons and possible advantages, e.g. the lack of enough data and choosing good priors, given in the answers seem great in theory, but still do not answer the question by providing, at least, one example. $\endgroup$
    – MNLR
    Apr 26, 2018 at 8:57
  • 1
    $\begingroup$ Circa 2017, there is now no question that deep learning outperforms all other methods in NLP. Long live transformers! $\endgroup$
    – profPlum
    Dec 11, 2022 at 16:18

9 Answers 9

38
$\begingroup$

One of the areas where Bayesian approaches are often used, is where one needs interpretability of the prediction system. You don't want to give doctors a Neural net and say that it's 95% accurate. You rather want to explain the assumptions your method makes, as well as the decision process the method uses.

Similar area is when you have a strong prior domain knowledge and want to use it in the system.

$\endgroup$
2
22
$\begingroup$

Bayesian networks and neural networks are not exclusive of each other. In fact, Bayesian networks are just another term for "directed graphical model". They can be very useful in designing objective functions neural networks. Yann Lecun has pointed this out here: https://plus.google.com/+YannLeCunPhD/posts/gWE7Jca3Zoq.

One example.

The variational auto encoder and derivatives are directed graphical models of the form $$p(x) = \int_z p(x|z)p(z) dz.$$ A neural networks is used to implemented $p(x|z)$ and an approximation to its inverse: $q(z|x) \approx p(z|x)$.

$\endgroup$
5
  • $\begingroup$ Can the two parts be trained jointly? $\endgroup$
    – nn0p
    Jul 28, 2016 at 14:50
  • $\begingroup$ Yes, that's what's typically done. $\endgroup$
    – bayer
    Nov 23, 2019 at 22:11
  • $\begingroup$ @bayer Not sure who to ask - do Bayesian networks encompass node PDFs in a way that neural networks cannot (or have e.g. assume Gaussians/delta-functions)? $\endgroup$
    – jtlz2
    Jan 6, 2020 at 11:10
  • $\begingroup$ @bayer PS Link is dead $\endgroup$
    – jtlz2
    Jan 6, 2020 at 11:11
  • $\begingroup$ Neural networks are just a computational architecture, they can represent anything. The link is dead because Google shutdown Plus. I don't have the quote anymore, sorry. $\endgroup$
    – bayer
    Jan 13, 2020 at 7:04
21
$\begingroup$

Sometimes you care as much about changing the outcome as predicting the outcome.

A neural network given enough training data will tend to predict the outcome better, but once you can predict the outcome, you then may wish to predict the effect of making changes in the input features on the outcome.

An example from real life, knowing that someone is likely to have a heart attack is useful, but being able to tell the person that if they stopped doing XX, the risk would reduce by 30% is of much greater benefit.

Likewise for customer retention, knowing why customers stop shopping with you, is worth as much as predicting the customers that are likely to stop shopping with you.

Also a simpler Bayesian Network that predicts less well but leads to more action being taken may often be better than a more “correct” Bayesian Network.

The biggest advantage of Bayesian networks over neural networks is that they can be used for causal inference . This branch is of fundamental importance to statistics and machine learning and Judea Pearl has won the Turing award for this research.

$\endgroup$
2
  • $\begingroup$ But neural networks can also be used to determine the role and importance of different features, right? $\endgroup$
    – Hossein
    Aug 31, 2019 at 12:50
  • $\begingroup$ Importance, sure, role, not really. A non-Bayesian NN essentially learns correlations between features and the output values. This is not sufficient to prove there's a direct causal link. $\endgroup$
    – jkm
    Feb 29, 2020 at 22:56
17
$\begingroup$

Excellent answers already.

One domain which I can think of, and is working extensively in, is the customer analytics domain.

I have to understand and predict the moves and motives of the customers in order to inform and warn both the customer support, the marketing and also the growth teams.

So here, neural networks do a really good job in churn prediction, etc. But, I found and prefer the Bayesian networks style, and here are the reasons for preferring it:

  1. Customers always have a pattern. They always have a reason to act. And that reason would be something which my team has done for them, or they have learnt themselves. So, everything has a prior here, and in fact that reason is very important as it fuels most of the decision taken by the customer.
  2. Every move by the customer and the growth teams in the marketing/sales funnel is cause-effect. So, prior knowledge is vital when it comes to converting a prospective lead into a customer.

So, the concept of prior is very important when it comes to customer analytics, which makes the concept of Bayesian networks very important to this domain.


Suggested Learning:

Bayesian Methods for Neural Networks

Bayesian networks in business analytics

$\endgroup$
7
$\begingroup$

Bayesian networks might outperform Neural Networks in small data setting. If the prior information is properly managed via the network structure, priors and other hyperparameters, it might have an edge over Neural Networks. Neural Networks, especially the ones with more layers, are very well known to be data hungry. Almost by definition lots of data is necessary to properly train them.

$\endgroup$
6
$\begingroup$

I've posted this link on Reddit and got a lot of feedback. Some have posted their answers here, others didn't. This answer should sum the reddit post up. (I made it community wiki, so that I don't get points for it)

$\endgroup$
2
$\begingroup$

Bayesian networks are preferred for genome interpretation. See, for example, this dissertation discussing computational methods for genome interpretation.

$\endgroup$
1
  • 2
    $\begingroup$ Why are they preferred? $\endgroup$ Jan 18, 2016 at 14:13
2
$\begingroup$

I did a small example for this once. From that, I think Bayesian Networks are preferred if you want to capture a distribution but your input training set doesn't cover the distribution well. In such cases, even a neural network that generalised well would not be able to reconstruct the distribution.

$\endgroup$
0
$\begingroup$

All of them, given a sufficiently small dataset. Proof: Consider a dataset with 0 data points. A neural network can't be fit to this dataset or provide predictions, whereas a Bayesian network can, assuming you set proper priors on all parameters; so the Bayesian network will do better. For small but nonzero datasets, priors are still very helpful.

Given a sufficiently large dataset, a neural network will eventually perform at least as well as a Bayesian network, since it will converge to the correct answer (thanks to the universal approximation theorem). But Bayesian nonparametrics are also a thing, so picking the right Bayesian network also guarantees convergence to the right answer. Moreover, "sufficiently large" can be really big, requiring way more data than you could ever realistically get your hands on. Neural networks exist because they can be trained quickly on large datasets, not because they're better at learning from the same dataset--the reality is just the opposite. If it weren't for the internet giving us absolutely insane amounts of data for free, we really wouldn't be using neural nets at all.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.