0
$\begingroup$

I have a database of text files and I would like to classify each section(i.e references, abstract, etc...) in the text.

For example:

Abstract Indexes: 100-300

References Indexes: 9000-10000

Is there a package that is doing something like that?

Is there a recommended approach for doing so?

$\endgroup$
3
  • $\begingroup$ What kind of text format is underlying. Has it some level of structure (like LaTeX) or is it free text? $\endgroup$
    – El Burro
    Commented Jun 2, 2017 at 14:26
  • $\begingroup$ It's free text. It's basically a text extracted from documents(docx, pdf etc...) so I have the source files also. $\endgroup$
    – Montoya
    Commented Jun 2, 2017 at 14:47
  • $\begingroup$ Use a sequence model like a CRF or RNN to classify the sections collectively; cf. structured prediction and sequence prediction. $\endgroup$
    – Emre
    Commented Jun 2, 2017 at 17:01

2 Answers 2

1
$\begingroup$

You might try to use the metadata of the individual sections for classification:

Examples:

  1. Length -> Abstracts are usually short.
  2. Existence of Figures -> Do not usually appear in abstract of references
  3. Relative Abundance of quotations. The references should have a significant higher share of reference keys than the others.

You could then use the metadata as input for any supervised learning approach. For example a Neural Network or a (boosted) decision tree.

$\endgroup$
0
$\begingroup$

A simple approach might use document vectors or LDA representing the text of each section (possibly chunking longer sections) with a simple classifier trained on those vectors to predict section. These can be found in gensim or sklearn.

A harder to implement but likely more powerful technique would be to train a recurrent neural network on the text of each section (again likely breaking it into smaller chunks) so that the representation your model learns is learned specifically to predict Abstract/Indices/etc ...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.