4
$\begingroup$

I need to extract relevant key phrases from a single document. Since I don't have a lot of documents, TF-IDF doesn't really work.

Currently I'm using TextRank. It produces okay-ish result - some really good phrases along with a lot of garbage.

Is there a better algorithm to use for this? Can anyone give me a rundown of available options?

Real-world use case: I'm developing a help desk app that comes with Knowledge Base (a bunch of articles, think of it as FAQ). When a user writes a new support ticket I want to extract key phrases and find the most relevant KB articles. Overall there is not enough data to train a model. I need to compare sets of key phrases I think.

$\endgroup$

1 Answer 1

3
$\begingroup$

A related keyword to your case can be Single Document Keyword Extraction. A good paper about this is:

We present a new keyword extraction algorithm that applies to a single document without using a corpus. Frequent terms are extracted first, then a set of cooccurrence between each term and the frequent terms, i.e., occurrences in the same sentences, is generated. Co-occurrence distribution shows importance of a term in the document as follows. If probability distribution of co-occurrence between term a and the frequent terms is biased to a particular subset of frequent terms, then term a is likely to be a keyword. The degree of biases of distribution is measured by the $\chi^2$-measure. Our algorithm shows comparable performance to tfidf without using a corpus.

You can find the paper here.

In sum, this paper gives a rank on keywords based on the defined $\chi^2$-measure.

$\endgroup$
3
  • $\begingroup$ Thanks. Do you know if there are any Python implementations? I don't think I'm smart enough to do it myself. $\endgroup$ Nov 22, 2017 at 16:39
  • $\begingroup$ @MaxAlFarakh airpair.com/nlp/keyword-extraction-tutorial $\endgroup$
    – trollster
    Nov 22, 2017 at 16:58
  • $\begingroup$ @trollster I tried RAKE. Did not produce good results. I'll look into it again. $\endgroup$ Nov 22, 2017 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.