1
$\begingroup$

Given that I have a word that does not occur in any of my documents: newword, and given that I have two classes: class1 and class2. For instance, the total number of words in class1 is 3, the total number of words in class2 is 6, and the number of unique words in all documents is 8.

Also, given the Naive Bayes formula, the word newword will have a higher probability of belonging to class1 (because the denominator will be lower while the numerator stays the same). Is there any statistical/logical theoretical explanation (that drives the algorithm) about this behavior?

In a nutshell, I just want to know what is the motivation of giving higher probability to classes with fewer words.

Ps.: I am using Laplace Smoothing. Therefore:
P(newword|class1) = 0 + 1 / 3 + 8 = 0.09 >
P(newword|class2) = 0 + 1 / 6 + 8 = 0.07

$\endgroup$
2
$\begingroup$

The Naive Bayes formula doesn't do what you say. For classes $C_k$ and words $x_1, \ldots, x_n$, the formula is

$P(C_k | x_1, \ldots, x_n) \propto P(C_k) \prod_{i=1}^N P(x_i | C_k)$

If the word newword doesn't occur, we have no prior information about $P(x_i | C_k)$, so all these conditional probabilities should be equal. So the right-hand side is just proportional to $P(C_k)$, which is independent of the number of words in $C_k$.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I edited my question :) $\endgroup$ – almanegra Apr 26 '18 at 1:38
  • $\begingroup$ I do not understand why you think the conditional probabilities P(newword|class) are given by these formulas. Since newword doesn't occur in the data, and you haven't given any other prior information, it seems that you actually have no principled way to calculate the conditional probability. I, and I guess most people, would assign equal a priori conditional probabilities, as mentioned in my answer. $\endgroup$ – Dave Kielpinski Apr 26 '18 at 23:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.