Problem statement :
We have documents with list of words in them. Overall these documents are classified into 2 group (say, good quality vs bad)
doc1 = [w1,w2,w3,w4] doc2 = [w4,w3,w3,w4] doc3 = [w2,w4,w8,w1] doc4 = [w5,w4,w0,w9]
doc group -
good_grp = [doc2, doc1] bad_grp = [doc3, doc4]
Now we have to find out which words actually are important to make the document good vs bad ?
Idea 1: Merge all words from documents that belong to document group 1 into single document say (good quality doc) and other one being (bad quality doc) and calculate tf-idf score per doc; but in this case we lose information of document level words and now just see document group level word importance.
doc1 = [w1,w2,w3,w4] doc2 = [w4,w3,w3,w4] doc3 = [w2,w4,w8,w1] doc4 = [w5,w4,w0,w9] good_grp = [w1,w2,w3,w4,w4,w3,w3,w4] bad_grp = [w2,w4,w8,w1,w5,w4,w0,w9]
Can someone help me to direct to a better approach tf-idf or any other technique to solve this problem?