1
$\begingroup$

Given that I have a word that does not occur in any of my documents: newword, and given that I have two classes: class1 and class2. For instance, the total number of words in class1 is 3, the total number of words in class2 is 6, and the number of unique words in all documents is 8.

Also, given the Naive Bayes formula, the word newword will have a higher probability of belonging to class1 (because the denominator will be lower while the numerator stays the same). Is there any statistical/logical theoretical explanation (that drives the algorithm) about this behavior?

In a nutshell, I just want to know what is the motivation of giving higher probability to classes with fewer words.

Ps.: I am using Laplace Smoothing. Therefore:
P(newword|class1) = 0 + 1 / 3 + 8 = 0.09 >
P(newword|class2) = 0 + 1 / 6 + 8 = 0.07

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The Naive Bayes formula doesn't do what you say. For classes $C_k$ and words $x_1, \ldots, x_n$, the formula is

$P(C_k | x_1, \ldots, x_n) \propto P(C_k) \prod_{i=1}^N P(x_i | C_k)$

If the word newword doesn't occur, we have no prior information about $P(x_i | C_k)$, so all these conditional probabilities should be equal. So the right-hand side is just proportional to $P(C_k)$, which is independent of the number of words in $C_k$.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I edited my question :) $\endgroup$
    – almanegra
    Commented Apr 26, 2018 at 1:38
  • $\begingroup$ I do not understand why you think the conditional probabilities P(newword|class) are given by these formulas. Since newword doesn't occur in the data, and you haven't given any other prior information, it seems that you actually have no principled way to calculate the conditional probability. I, and I guess most people, would assign equal a priori conditional probabilities, as mentioned in my answer. $\endgroup$
    – Dave Kielpinski
    Commented Apr 26, 2018 at 23:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.