3
$\begingroup$

Here is my understanding of the difference between a Bernoulli and a Multinomial Naive Bayes:

Bernoulli explicitly models the presence/absence of a feature, whereas Multinomial doesn't.

  • Is there something that I am missing?
  • Can someone explain why this difference matters intuitively? Perhaps using an example where you would obviously use one instead of the other

On the wikipedia page, they say:

Note that a naive Bayes classifier with a Bernoulli event model is not the same as a multinomial NB classifier with frequency counts truncated to one.

Why is that?

$\endgroup$
3
$\begingroup$

Bernoulli models the presence/absence of a feature. Multinomial models the number of counts of a feature. Here's a concise explanation.

Wikipedia warns that

Note that a naive Bayes classifier with a Bernoulli event model is not the same as a multinomial NB classifier with frequency counts truncated to one.

To understand why, we should note that, as this page nicely explains,

Whereas the binomial distribution generalises the Bernoulli distribution across the number of trials, the multinoulli distribution generalises it across the number of outcomes, that is, rolling a dice instead of tossing a coin.

What does this imply for us? Multinomial NB cares about counts for multiple features that do occur, whereas Bernoulli NB cares about counts for a single feature that do occur and counts for the same feature that do not occur.

This means that, for example, Multinomial NB will classify a document based on the counts it finds of multiple keywords; whereas Bernoulli NB can only focus on a single keyword, but will also count how many times that keyword does not occur in the document.

So they do model slightly different things. If you have discrete multiple features to worry about, you have to use Multinomial NB. But if you only have a single feature to worry about, then you can make a modelling choice based on the above.

$\endgroup$
  • $\begingroup$ Thanks for the link. I believe I understand the difference. What I struggle with is finding a use case where it would make more sense to use one instead of the other? Any idea for that? $\endgroup$ – Valentin Calomme Feb 13 '18 at 15:49
  • $\begingroup$ Multinomial is more general, and hence more powerful. Even in statistics, the Multinomial distribution is a generalisation of the Bernoulli distribution (with parameters $k=2,n=1$). This extra power means more parameters to estimate but should compensate for that. $\endgroup$ – A. G. Feb 13 '18 at 15:52
  • $\begingroup$ Alright, then, I am not sure I understand what they say on the NB wikipedia page: "Note that a naive Bayes classifier with a Bernoulli event model is not the same as a multinomial NB classifier with frequency counts truncated to one.". Shouldn't the probabilities be modeled the same way? $\endgroup$ – Valentin Calomme Feb 14 '18 at 8:00
  • 1
    $\begingroup$ Good question. I've edited my answer to address it. $\endgroup$ – A. G. Feb 14 '18 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.