5
$\begingroup$

I'm doing a keyword extraction using TF-IDF on a large number of documents. Currently, I'm splitting each sentence based on n-gram. More particularly, I'm using trigrams. However, this is not the best way to split each sentence into ints constituting keywords. For example a noun phrase like 'triple heart bypass' may not always get detected as one term.

The other alternative to chunk each sentence into its constituting elements look to be part of speech tagging and [chunking][1] in [Open NLP][2]. In this approach phrase like 'triple heart bypass' always gets extracted as a whole but the downside is in TF-IDF the frequency of extracted terms (phrases) dramatically drops.

Does anyone have any suggestion on either or these two approaches or have any other ideas to improve the quality of the keywords?

Improve this question
$\endgroup$
3
  • $\begingroup$ Word2vec has an interesting option that identifies phrases for you, you could try that. $\endgroup$
    – jamesmf
    Commented Nov 19, 2015 at 22:30
  • $\begingroup$ Are you fine with just using an API / service? $\endgroup$
    – Adam Bittlingmayer
    Commented Jan 19, 2016 at 16:16
  • $\begingroup$ If you are using phrases then keyword extraction is maybe not the best title $\endgroup$
    – paparazzo
    Commented Jan 21, 2016 at 21:38

1 Answer 1

Reset to default
1
$\begingroup$

I think the tagging approach has some merit here. The frequency drop you're observing as a consequence of using this is to be expected, I think. After all, keywords are the words that help differentiate a document from others in a corpus. If you have access to an ontology related to the subject matter of your corpus, you could try to map your rare keyword tags to the ontology, and use parent-level information for each to get a slightly more general set of keyword tags! If this is an approach that interests you, Stanford's open source protégé system is a good framework for working on this.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.