6
$\begingroup$

Working with the Naive Bayes spam filtering article on Wikipedia (https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering)

enter image description here

enter image description here

Is it a binary multinomial equation or a Bernoulli form or some other?

In this 2006 paper, it discusses that there are many Naive Bayes algorithms: Spam Filtering with Naive Bayes – Which Naive Bayes? http://www.aueb.gr/users/ion/docs/ceas2006_paper.pdf

The paper states that binary Multinomial NB performs best.

If it is not multinomial, what changes are needed to make it so?

$\endgroup$
  • $\begingroup$ Your question is not clear! What do you mean by "if it is not multinomial, what changes are needed to make it so?" $\endgroup$ – eliasah Sep 23 '15 at 17:33
  • $\begingroup$ What differences are there, if any, in the above formulas between multinomial and multivariate Naive Bayes? $\endgroup$ – Chris Sep 24 '15 at 13:11
  • 1
    $\begingroup$ Multinomial is used for discrete variables and the multivariate normal is used for continuous variable! $\endgroup$ – eliasah Sep 24 '15 at 13:16
  • $\begingroup$ So given the formulas, which are they? $\endgroup$ – Chris Sep 24 '15 at 17:22
6
$\begingroup$

First off, yes there are different Naive Bayes algorithms. But they are all based on the same principle, namely Bayes Theorem where the features are assumed to be independent.

Here is a short guide on when to use which for spam detection (or document classification in general for that matter):

  • Bernoulli Naive Bayes

    Each e-mail is represented as a binary vector, and in this case, whether a word is present in the e-mail matters as much as if it isn't. For instance, the fact that "viagra" is in the e-mail might mean it's "spam" and the fact it isn't might mean it's "not spam"

  • Multinomial Naive Bayes with boolean features

    Each e-mail is represented as a binary vector, and in this case, whether a word is present in the e-mail matters more than if it isn't. For instance, if "grandma" is in the e-mail, it might mean that it's "not spam", but if it isn't in the e-mail, it doesn't necessarily mean that it's "spam".

  • Multinomial Naive Bayes with term frequency

    Each e-mail is represented by how many times each word occurs.

  • Gaussian Naive Bayes

    This deals with continuous values, so this isn't applicable here.

$\endgroup$
2
$\begingroup$

Both screengrabs you've posted are not associated with any particular method of calculating probabilities. So they keep the formulae for calculating Naive Bayes very general, without endowing the notation with any specific meaning.

To make them "anything", whether some variant of Bernoulli, Multinomial or so on, involves defining how the $P(...)$s or $p$s are defined and evaluated.

Since you want to use this paper, you can simply "plug in" formulae to evaluate them as defined in the paper.

For example, for multivariate Bernoulli Naive Bayes with a Laplacian prior, you would plug in:

enter image description here

Similarly, for Multinomial Naive Bayes (whether using TF or binary attributes), you would plug in:

enter image description here

As for the second formula, $p = \frac{1}{1+e^{\eta}}$, it is simply a more computationally-accurate way to evaluate the general formula Naive Bayes posterior calculation $p = \frac{\prod_{i=1}^N p_i}{\prod_{i=1}^N p_i + \prod_{i=1}^N (1-p_i)}$. This is because performing summation operations in the log space over many operands may often be more accurate than performing multiplication operations in the usual space, due to the way floating-point arithmetic is implemented in computers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.