20
$\begingroup$

Are there any articles or discussions about extracting part of text that holds the most of information about current document.

For example, I have a large corpus of documents from the same domain. There are parts of text that hold the key information what single document talks about. I want to extract some of those parts and use them as kind of a summary of the text. Is there any useful documentation about how to achieve something like this.

It would be really helpful if someone could point me into the right direction what I should search for or read to get some insight in work that might have already been done in this field of Natural language processing.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
26
$\begingroup$

What you're describing is often achieved using a simple combination of TF-IDF and extractive summarization.

In a nutshell, TF-IDF tells you the relative importance of each word in each document, in comparison to the rest of your corpus. At this point, you have a score for each word in each document approximating its "importance." Then you can use these individual word scores to compute a composite score for each sentence by summing the scores of each word in each sentence. Finally, simply take the top-N scoring sentences from each document as its summary.

Earlier this year, I put together an iPython Notebook that culminates with an implementation of this in Python using NLTK and Scikit-learn: A Smattering of NLP in Python.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ Yes, that would probably be it. I could also add additional weights to some words, that I already know that are informative. Thanks for your help and useful links. $\endgroup$
    – MaticDiba
    Commented Dec 9, 2014 at 9:33
  • $\begingroup$ So can i use this on a pdf? :) $\endgroup$
    – Adam
    Commented Apr 14, 2017 at 22:25
  • $\begingroup$ Yes, you can use this on the text in a PDF, assuming you've already extracted the plain text from the PDF using something like pdftotext. $\endgroup$
    – Charlie Greenbacker
    Commented Apr 15, 2017 at 23:26
1
$\begingroup$

Lots of keyword extraction techniques out there depend on factors like:

  1. Grammatical quality of text
  2. Length of text
  3. Whether you are looking for a single keyword or phrasal keyword etc.

But in general, if you have a long text and you want to extract keywords automatically from that, I would recommend you to go through follow articles:

  1. TextRank

  2. RAKE [Rapid Automatic Keyword Extraction]

  3. Topica

Also to extract custom (special) keywords which is not coming through the above techniques, have a look at the post below:

Extract Custom Keywords using NLTK POS tagger in python

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.