0
$\begingroup$

I have a set of documents where I need to extract important keywords in the document and then rank those keywords.

The ranking should be done based on relevance and/or other metrics. Are there any popular algorithms/methods/libraries that I can use for this task?

Improve this question
$\endgroup$
2
  • $\begingroup$ What is an "Important" keyword in the documents? $\endgroup$
    – Thomas Cleberg
    Commented Dec 5, 2017 at 14:27
  • $\begingroup$ @ThomasCleberg Hi, what I meant by importance is the kind of concepts/terms that represent the document and relevance indicates some ordering of the keywords. $\endgroup$
    – Volka
    Commented Dec 5, 2017 at 22:16

2 Answers 2

Reset to default
2
$\begingroup$

It would help if you clarified what "relevance" and "important" mean, but you should take a look at Term Frequency-Inverse Document Frequency.

tf-idf weights words by the frequency of appearance within a document but penalizes words that appear across many documents. The concept is that words appearing many times in a single document are likely noteworthy, but they could also just be common words that appear frequently in all documents.

tf-idf has implementations in most popular langauges including Python and R. The Python Implementation is in Sci-kit learn and is called TfidfVectorizer. It takes a list of filenames or a single document as input. You can also specifiy the stop words to apply to the documents and allows for tokenization. You probably also would want to apply a stemmer to your documents which can be done using Sci-kit learn. If you prefer to work in R, you would want to use the tm package. This package has numerous functions for text mining.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Hi, thanks for the answer. I tried tf-idf. But it did not work for my dataset.Are there any other algorithms that I can use instead of tf-idf? $\endgroup$
    – Volka
    Commented Dec 5, 2017 at 22:14
  • $\begingroup$ Can you give us some more info about your problem? A sample document and the ranking you are looking for would be helpful. $\endgroup$
    – willk
    Commented Dec 6, 2017 at 13:04
  • $\begingroup$ for example, the concepts such as natural language processing, data mining, data visualisation should rank in the top, where as unnessary terms such as methods, techniques, main issue, most important should rank lower. $\endgroup$
    – Volka
    Commented Dec 6, 2017 at 13:12
  • $\begingroup$ Thanks for the example. I am not aware of any algorithm that would be able to automatically detect which words are important concepts and which are unnecessary unless you provided the list of important words ahead of time. In that case, you could just provide a list of the important words and count the number of occurrences in each document. $\endgroup$
    – willk
    Commented Dec 6, 2017 at 19:17
  • $\begingroup$ Oh :( please let me know if you could find anything. $\endgroup$
    – Volka
    Commented Dec 6, 2017 at 23:04
0
$\begingroup$

API models exist which can achieve this.

https://askmacgyver.com/explore/program/universal-topic-classifier/5S2Q5x8K

It takes an array of categories or "bag of words" and a text string to analyze. It then returns a sorted percentage of relevance for this provided keywords.

Input Data

  {
    "text": "this bank provides an excelent service to its clients when opening a new account and with other operations",
    "classes": [
        "bank account",
        "online banking",
        "technical support",
        "mortgage",
        "retirement savings",
        "mutual funds",
        "student loan",
        "credit card",
        "financial news"
    ],
    "minCutOff": "0.001"
}

API Response

{
    "bank account": 0.6448822158372491,
    "technical support": 0.40099627067600924,
    "financial news": 0.28635987039897565,
    "mortgage": 0.2676284175575462,
    "student loan": 0.257628495744561,
    "online banking": 0.32395217514082025,
    "credit card": 0.2144582134037077,
    "mutual funds": 0.09250890827081894,
    "retirement savings": 0.13690496892541437
}
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.