Naive Bayes for Multi label text classification

Question

How to use Naive Bayes for multi-label text classification in R.

I tried using naiveBayes() from e1071 library but it seems that while training, it doesn't accept multi-label class variable.

I created TermDocumentMatrix using the text document corpus and tried to create the model using this matrix and class variable(list of topics a particular document belongs to). Below is the code that I used.

trainvector <- as.vector(traindata$bodyText)

trainsource <- VectorSource(trainvector)

traincorpus <- Corpus(trainsource)

trainmatrix <- t(TermDocumentMatrix(traincorpus))

model <- naiveBayes(as.matrix(trainmatrix), as.factor(traindata$topics))

The last line gives below error:

Error in sort.list(y) : 'x' must be atomic for 'sort.list'

Have you called 'sort' on a list?

I tried using

model <- naiveBayes(as.matrix(trainmatrix), as.factor(unlist(traindata$topics)))

but got error:

Error in tapply(var, y, mean, na.rm = TRUE) : 
  arguments must have same length

Answer 1

Since generally, Naive bayes alogorithms implemented by most libraries don't support multilabel classification. Although You can devise your own algorithm taking inspirations from naive bayes approach.

For a particular class c, and document d (with 3 words w1,w2,w3)

'/' stands for 'given' p(a/b) = probability(a given b)

As per bayes theorem.

p(c/d) = p(c,d) / p(d)

p(c/d) = (p(c)*p(d/c)) / p(d)

where:

p(d/c) = p(w1/c)*p(w2/c)*p(w3/c)

Since words are assumed to be independent of each other.

And p(w1/c) can be obtained using your code, calculating count of w1 inside class c documents divided by overall count of w1 in all documents or you can use your own logic

But If want to avoid writing detailed code

you can restructure your input data to achieve multilabel classification. such that a given document d with n labels/classes (eg d labeled with c1,c2,c3) get expanded into data of n samples (3 here) of same document d with different label each time ((d,c1),(d,c2),(d,c3)). Now you can pass this data into any R/python library supporting multinomial naive bayes. Same needs to be done in your dataset as well. Currently, you are passing traindata$topics which is y variable (training labels) as it is, without modification.

Even after you train your data using this approach. You need to use a probability threshold e.g 0.2, so that class labels with probability above 0.2 will be assigned to that test document.

More better Approach which requires restructuring your input

If you have 'n' class labels then you can train 'n' different binary naive bayes classifier for each of the class. For example, for training a classifier for class 'c1', you train a naive bayes classifier with dependent variable Y denoting the presence of class c1 on that document as '1' and absence as '0'.

After you train 'n' binary naive bayes classifier using this approach. You will now use the output of these n classifiers. e.g if out of these, n classifiers, if a particular classifier, which corresponds to class 'c1' has output probability above 0.5, then class label 'c1' will be assigned to this test document.