How to rank documents using Bag of words approach

Question

I want to cluster the documents I get for Google scholar search using the Bag of words model. I thought of using Java as the language.

Assume for the keyword k, Google scholar gives me 50 results. If I have a predefined set of words w1, w2, w3... wn, how can I rank the the documents which have the predefined set of words most? How can I apply Bag of words model for this? Do I need a clustering algorithm like k-means? And do I need to perform NLP techniques as well?

Say that the word w1 has several synonyms. How can I consider those synonyms as well for document ranking? Will I have to create a corpus containing all the abbreviations, synonym etc for that?

Are there any good tutorials available for this? Will choosing Java over Python will be an disadvantage since most of the resources (Ex - Scikit) were in Python?

Answer 1

If you're just looking to rank documents according to how many appearances your words w1,..,wn contain, then there's no need for clustering or machine learning in general: Clustering your 50 results will give you a partition of these results into clusters containing results that are similar to one another and different from the results in other clusters.

If you just need to rank by word occ, just count the frequencies of your words in each document (including synonyms, which you can get e.g. from Wordnet automatically if you prefer) and sum them up.

Answer 2

If you are just looking to rank documents, @Sharon answer is what you need (+1).

After you have ranked them, indeed the next logical step would be to cluster the documents, to find out which of them are similar. In particular, if your list of words is heterogeneous, the rank by itself might not give you very clear information. This is quite a good python notebook, everything is kept simple but at the same time it explains all the steps you should follow.

Document clustering in Python

If you just wanted to detect the topics of your document (i.e. without your list of words), this is a good SO answer, which includes links to resources.

As for your question about Java v. Python, I cannot tell you for sure that Java would be a disadvantage. Even though all the links I provide are in python, I'm certain there's quite a few nl resources in Java. However I don't know them well and I cannot compare.