1
$\begingroup$

In short: For naive Bayes and text classification, do you multiply a probability for each instance of a word in a document or once if the word occurs?

In more detail: The question is how to calculate naive Bayes for a document. We have a corpus of text from which we can calculate the frequency of documents with class $C$ having a word $w$. We then calculate the probability $P(w_1|C)...P(w_n|C)$ and choose the $C$ that maximizes this number. But I'm not sure whether we multiply a probability for each word whenever it occurs or just once if it occurs in the document? Like, if the text is "$w_1 \ w_2 \ w_1$", is the estimated probability $P(w_1|C)^2 P(w_2|C)$ or is it $P(w_1|C)P(w_2|C)$?

$\endgroup$
1
$\begingroup$

You could use either of the two approaches:

  • Just count whether your documents in your corpus contain or not a word $w$ and then estimate the likelihoods of $P(w_1|C)...P(w_n|C)$, as you describe

  • You could count how many instances of any word has each of the documents of your corpus, but then the estimation of the likelihoods becomes a bit more complicated than what you describe; in essence you need to use a multinomial distribution: $P(w_1|C=k) = \sum_{i=1}^{N}x_{it}z_{ik}/\sum_{s=1}^{|V|}\sum_{i=1}^{N}x_{is}z_{ik}$, where $N$ is the number of documents, $x_{it}$ in the numerator is the counts of word $w_1$ per document, $s$ in the denominator iterates the number of terms $|V|$ in your corpus and $x_{is}$ is the number times each term appears in each document, and finally $z_{ik}$ is 1 if the document belongs to class k, 0 otherwise, in both numerator in the denominator

Hopefully the short explanation makes sense, anyway you have a very complete and useful reference here (the same notation in the formula above).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.