Predicting New Data with Naive Bayes

Question

Say I had the following training set for a Naive Bayes algorithm.

   Outlook    Person    Play Golf?
   -------    ------    ----------
   Sunny      Joe       Yes
   Sunny      Mary      Yes
   Raining    Joe       Yes
   Raining    Mary      No
   Raining    Harry     Yes

If try to predict whether Harry will play golf on a sunny day (which I have no data for). Would it be correct to exclude the person attribute and use the remaining outlook attribute to calculate the probability of this happening? Or could that potentially cause problems with a larger data set that I'm unaware of?

Answer 1

Among Naive Bayes assumptions the main one is that features are conditionally independent. For our problem we would have:

$$P(Play|Outlook,Person) \propto P(Play)P(Outlook|Play)P(Name|Play)$$

To address question is Harry going to play on a sunny day?, you have to compute the following:

$$P(Yes|Sunny,Harry) = P(Yes)P(Sunny|Yes)P(Harry|Yes)$$ $$P(No|Sunny,Harry) = P(No)P(Sunny|No)P(Harry|No)$$

and choose the probability with bigger value.

That is what theory says. To address your question I will rephrase the main assumption of Naive Bayes. The assumptions that features are independent given the output means basically that the information given by joint distribution can be obtained by product of marginals. In plain English: assume you can find if Harry plays on sunny days if you only know how much Harry plays in general and how much anybody plays on sunny days. As you can see, you simply you would not use the fact that Harry plays on sunny days even if you would have had that record in your data. Simply because Naive Bayes assumes there is no useful information in the interaction between the features, and this is the precise meaning of conditional independence, which Naive Bayes relies upon.

That said if you would want to use the interaction of features than you would have either to use a different model, or simply add a new combined feature like a concatenation of factors of names and outlook.

As a conclusion when you do not include names in your input features you will have a general wisdom classifier like everybody plays no matter outlook, since most of the instances have play=yes. If you include the name in your input variables you allow to alter that general wisdom with something specific to player. So your classifier wisdom would look like players prefer in general to play, no matter outlook, but Marry like less to play less on Rainy.

There is however a potential problem with Naive Bayes on your data set. This problem is related with the potential big number of levels for variable Name. In order to approximate the probability there is a general thing that happens: more data, better estimates. This probably would happen with variable Outlook, since there are two levels and adding more data would probably not increase number of levels. So the estimates for Outlook would be probably better with more data. However for name you will not have the same situation. Adding more instances would be possible perhaps only by adding more names. Which means that on average the number of instances for each name would be relatively stable. And if you would have a single instance, like it is the case for Harry, you do not have enough data to estimate $P(Harry|No)$.

As it happens this problem can be alleviated using smoothing. Perhaps Laplace smoothing (or a more general for like Lindstone) is very helpful. The reason is that estimates based on maximum likelihood have big problems with cases like that.

I hope it answers at least partially your question.