9
$\begingroup$

Problem

I have tried using Naive bayes on a labeled data set of crime data but got really poor results (7% accuracy). Naive Bayes runs much faster than other alogorithms I've been using so I wanted to try finding out why the score was so low.

Research

After reading I found that Naive bayes should be used with balanced datasets because it has a bias for classes with higher frequency. Since my data is unbalanced I wanted to try using the Complementary Naive Bayes since it is specifically made for dealing with data skews. In the paper that describes the process, the application is for text classification but I don't see why the technique wouldn't work in other situations. You can find the paper I'm referring to here. In short the idea is to use weights based on the occurences where a class doesn't show up.

After doing some research I was able to find an implementation in Java but unfortunately I don't know any Java and I just don't understand the algorithm well enough to implement myself.

Question

where I can find an implementation in python? If that doesn't exist how should I go about implementing it myself?

$\endgroup$
5
$\begingroup$

Naive Bayes should be able to handle imbalanced datasets. Recall that the Bayes formula is

$$P(y \mid x) = \cfrac{P(x \mid y) \, P(y)}{P(x)} \propto P(x \mid y) \, P(y)$$

So $P(x \mid y) \, P(y)$ takes the prior $P(y)$ into account.

In your case maybe you overfit and need some smoothing? You can start with +1 smoothing and see if it gives any improvements. In python, when using numpy, I'd implement the smoothing this way:

table = # counts for each feature 
PT = (table + 1) / (table + 1).sum(axis=1, keepdims=1)

Note that this is gives you Multinomial Naive Bayes - which applies only to categorical data.

I can also suggest the following link: http://www.itshared.org/2015/03/naive-bayes-on-apache-flink.html. It's about implementing Naive Bayes on Apache Flink. While it's Java, maybe it'll give you some theory you need to understand the algorithm better.

$\endgroup$
  • $\begingroup$ You may be correct it looks like naive Bayes only has the imbalance problem with text classification. Can you explain what smoothing is? $\endgroup$ – grasshopper Jun 21 '15 at 15:34
  • $\begingroup$ Have you had a look at the link? It's explained there. In short, when estimating $P(x \mid y)$ sometimes $x$ is never seen during training and smoothing ensures that it doesn't ruin the classifier performance by adding some extra count (in case of laplace or +1 smoothing) to all the features. $\endgroup$ – Alexey Grigorev Jun 21 '15 at 17:35
0
$\begingroup$

My implementation of Complement Naive Bayes in scikit-learn can be found here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.