# How to approach TF-IDf based analysis?

Problem statement :

We have documents with list of words in them. Overall these documents are classified into 2 group (say, good quality vs bad)

docs -

doc1 = [w1,w2,w3,w4]
doc2 = [w4,w3,w3,w4]
doc3 = [w2,w4,w8,w1]
doc4 = [w5,w4,w0,w9]


doc group -

good_grp = [doc2, doc1]


Now we have to find out which words actually are important to make the document good vs bad ?

Idea 1: Merge all words from documents that belong to document group 1 into single document say (good quality doc) and other one being (bad quality doc) and calculate tf-idf score per doc; but in this case we lose information of document level words and now just see document group level word importance.

doc1 = [w1,w2,w3,w4]
doc2 = [w4,w3,w3,w4]
doc3 = [w2,w4,w8,w1]
doc4 = [w5,w4,w0,w9]

good_grp = [w1,w2,w3,w4,w4,w3,w3,w4]


Can someone help me to direct to a better approach tf-idf or any other technique to solve this problem?

I think here you must maintain the actual tf-idf and create corpus over it.. Assuming you already have lables for documents available. You can rum classification over it.

Best classification I am anticipating for this problem would be naive bayes..

• just clarifying, just applying tf-idf over all documents will give you list of words that are most important per document. But i didnt understand how we can run classification on it ? is it similar to this post ? towardsdatascience.com/… – gagan malhotra Jan 8 '20 at 19:35
• Yes, as per update it seems my suggested approach worked.. If so please consider it to mark "right"(button just below downvote). This will help other to find this post quickly. – SKB Jan 9 '20 at 1:58

A direct way to find the words which are the most representative of a class is to calculate the probability of the class given a word:

$$p(c|w)=\frac{\#\{\ d\ |\ label(d)=c\ \land w\in d\}}{\#\{\ d\ |\ w\in d\ \}}$$

Ranking the words according to their probability $$p(c|w)$$ gives:

• highest values: the most correlated words for the class
• lowest values: the least correlated words for the class

Remark: with this method it's safer to filter out the least frequent words (e.g. remove the words with frequency lower than 3), because these are likely to happen by chance so they are not really representative.

• Thanks for the answer @Erwan Just wanted to verify that i understand it correctly....so to calculate the probability of card given word =((Number of documents per given class of document) * (number of words in current document)) / (number of documents that has this word) – gagan malhotra Jan 8 '20 at 18:44
• @gaganmalhotra No I think there's a mistake in your formula. Let's say you want this probability for class "good" and word "hello": p(good|hello) = (number of "good" documents which contain "hello") / (number of all documents which contain "hello"). In other words it's just the proportion of "good" documents among all the documents which contain the word "hello". – Erwan Jan 8 '20 at 19:08
• Thanks for the help @Erwan! – gagan malhotra Jan 8 '20 at 19:16
• But in this approach we would miss out of term frequency of words appearing in the document, do you have any idea how we can incorporate that ? @Erwan – gagan malhotra Jan 8 '20 at 19:21
• @gaganmalhotra I don't think it's useful to take word frequency into account since the label is associated with a document, but if you want we can assume that every word in a document is labelled with the label of this doc. In this case the probability just becomes (frequency of w in any "good" document) / (total frequency of w in any document) (in other words: sum of the frequency of w across all the documents which are good divided by sum of the frequency of w across all the documents) – Erwan Jan 8 '20 at 20:47

Update:

One thing that worked the best with my data was converting the words into tf-idf vectors per document and applying Naive bayes on it to predict the probability per document or word.