When classifying documents with naive Bayes, which probabilities do you multiply in case of repeated terms?

Question

In short: For naive Bayes and text classification, do you multiply a probability for each instance of a word in a document or once if the word occurs?

In more detail: The question is how to calculate naive Bayes for a document. We have a corpus of text from which we can calculate the frequency of documents with class $C$ having a word $w$. We then calculate the probability $P(w_1|C)...P(w_n|C)$ and choose the $C$ that maximizes this number. But I'm not sure whether we multiply a probability for each word whenever it occurs or just once if it occurs in the document? Like, if the text is "$w_1 \ w_2 \ w_1$", is the estimated probability $P(w_1|C)^2 P(w_2|C)$ or is it $P(w_1|C)P(w_2|C)$?

Answer 1

You could use either of the two approaches:

• Just count whether your documents in your corpus contain or not a word $w$ and then estimate the likelihoods of $P(w_1|C)...P(w_n|C)$, as you describe

• You could count how many instances of any word has each of the documents of your corpus, but then the estimation of the likelihoods becomes a bit more complicated than what you describe; in essence you need to use a multinomial distribution: $P(w_1|C=k) = \sum_{i=1}^{N}x_{it}z_{ik}/\sum_{s=1}^{|V|}\sum_{i=1}^{N}x_{is}z_{ik}$, where $N$ is the number of documents, $x_{it}$ in the numerator is the counts of word $w_1$ per document, $s$ in the denominator iterates the number of terms $|V|$ in your corpus and $x_{is}$ is the number times each term appears in each document, and finally $z_{ik}$ is 1 if the document belongs to class k, 0 otherwise, in both numerator in the denominator

Hopefully the short explanation makes sense, anyway you have a very complete and useful reference here (the same notation in the formula above).