Rank terms in a bag -of-words model

I have a set of documents where I need to extract important keywords in the document and then rank those keywords.

The ranking should be done based on relevance and/or other metrics. Are there any popular algorithms/methods/libraries that I can use for this task?

• What is an "Important" keyword in the documents? Dec 5 '17 at 14:27
• @ThomasCleberg Hi, what I meant by importance is the kind of concepts/terms that represent the document and relevance indicates some ordering of the keywords. Dec 5 '17 at 22:16

It would help if you clarified what "relevance" and "important" mean, but you should take a look at Term Frequency-Inverse Document Frequency.

tf-idf weights words by the frequency of appearance within a document but penalizes words that appear across many documents. The concept is that words appearing many times in a single document are likely noteworthy, but they could also just be common words that appear frequently in all documents.

tf-idf has implementations in most popular langauges including Python and R. The Python Implementation is in Sci-kit learn and is called TfidfVectorizer. It takes a list of filenames or a single document as input. You can also specifiy the stop words to apply to the documents and allows for tokenization. You probably also would want to apply a stemmer to your documents which can be done using Sci-kit learn. If you prefer to work in R, you would want to use the tm package. This package has numerous functions for text mining.

• Hi, thanks for the answer. I tried tf-idf. But it did not work for my dataset.Are there any other algorithms that I can use instead of tf-idf? Dec 5 '17 at 22:14
• Can you give us some more info about your problem? A sample document and the ranking you are looking for would be helpful. Dec 6 '17 at 13:04
• for example, the concepts such as natural language processing, data mining, data visualisation should rank in the top, where as unnessary terms such as methods, techniques, main issue, most important should rank lower. Dec 6 '17 at 13:12
• Thanks for the example. I am not aware of any algorithm that would be able to automatically detect which words are important concepts and which are unnecessary unless you provided the list of important words ahead of time. In that case, you could just provide a list of the important words and count the number of occurrences in each document. Dec 6 '17 at 19:17
• Oh :( please let me know if you could find anything. Dec 6 '17 at 23:04

API models exist which can achieve this.

It takes an array of categories or "bag of words" and a text string to analyze. It then returns a sorted percentage of relevance for this provided keywords.

Input Data

  {
"text": "this bank provides an excelent service to its clients when opening a new account and with other operations",
"classes": [
"bank account",
"online banking",
"technical support",
"mortgage",
"retirement savings",
"mutual funds",
"student loan",
"credit card",
"financial news"
],
"minCutOff": "0.001"
}


API Response

{
"bank account": 0.6448822158372491,
"technical support": 0.40099627067600924,
"financial news": 0.28635987039897565,
"mortgage": 0.2676284175575462,
"student loan": 0.257628495744561,
"online banking": 0.32395217514082025,
"credit card": 0.2144582134037077,
"mutual funds": 0.09250890827081894,
"retirement savings": 0.13690496892541437
}