Extracting specific portions of text from poorly formatted document?

Question

I have a corpus of text files (free books) which are poorly formatted. The goal is to extract a particular chapter (say chapter 2) from the raw text with all weird formatting removed. Some documents are literal copy and pastes from pdfs, so that the text contents from a chapter would have page numbers interspersed throughout, etc., and I.e., it would be impossible to create a good regex rule.

I've manually gone through a number of documents and stripped out all unwanted formatting to arrive at the desired, clean section of text. Using this as the training set, what would be a good model to set up for this task?

To clarify, the input would be a (very long) string, and the output would also be a string.

EDIT:

Please note that this particular example can be very easily handled by a series of regex rules, but it's only for illustration purposes only. Imagine if each document had their own minor idiosyncrasies making it very difficult to come up with a general series of regex rules to apply to all docs in the corpus.

Sample input:

Book Title ABC

Table of Contents

Preface ................................ Pg 1
Chapter 1 .............................. Pg 2
Chapter 2 .............................. Pg 3
Chapter 3 .............................. Pg 4

Preface

<p> Contents for the preface sit here.  Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse scelerisque turpis ac commodo porttitor. Suspendisse iaculis lacinia ante, non suscipit purus venenatis et. </p>

- 1 -

Leftover Contents from Preface

Chapter 1

Praesent at justo sit amet nulla porttitor hendrerit. Etiam pretium sem ac feugiat posuere. Nulla mi risus, mollis sit amet ligula egestas, lacinia dapibus tellus. Proin iaculis ligula ultricies nunc elementum maximus. Nulla accumsan libero at nunc scelerisque, ut fermentum lacus euismod. Nulla imperdiet pharetra laoreet. Quisque id mollis diam. Vestibulum interdum orci at lectus ullamcorper pharetra. Curabitur laoreet mollis pharetra.

- 2 -

Chapter 2

Cras diam nibh, congue sit amet imperdiet quis, finibus imperdiet tellus. Sed imperdiet risus id elit consequat cursus. Curabitur consequat facilisis molestie. Ut varius urna vel ornare scelerisque. Nullam dictum tempus sapien, in pulvinar tellus consectetur non. Vestibulum consequat pretium iaculis.

- 3 -

Leftover contents from Chapter 2.  Vivamus pretium mauris at metus egestas, eget luctus ligula auctor. 

Chapter 3

Maecenas vel dapibus lorem. Sed eget justo sit amet libero aliquam maximus a quis augue. Nunc consequat, urna quis elementum condimentum, quam arcu consectetur tellus, vehicula gravida est purus id nisl.

- 4 -

Nulla eget molestie velit, pharetra euismod magna. Vestibulum ultricies justo vitae massa vulputate, et venenatis orci fringilla.

Index

Sample Output:

Cras diam nibh, congue sit amet imperdiet quis, finibus imperdiet tellus. Sed imperdiet risus id elit consequat cursus. Curabitur consequat facilisis molestie. Ut varius urna vel ornare scelerisque. Nullam dictum tempus sapien, in pulvinar tellus consectetur non. Vestibulum consequat pretium iaculis.  
Leftover contents from Chapter 2.  Vivamus pretium mauris at metus egestas, eget luctus ligula auctor. 

Answer 1

If I understand the problem, you have text that is pre-sliced into some unit of interest (your chapter), that requires pre-processing to convert into a form suitable for further work. You've done some of this pre-processing by hand and want to use it as a training set for a model to infer the rules that should be applied to accomplish the same result on raw text.

Because I don't know what you intend to do with the scrubbed text, I'm going to assume that it's destined for natural language processing to extract some semantic content. Here's what you'd normally do.

1. Convert unicode encoding to UTF8 to get rid of the diacritical marks and other characters that aren't part of your target language

2. Tokenize the text into a list of strings of words and punctuation marks

3. Strip (or optionally convert to words if short) numbers

4. Strip excess white space

5. Optionally strip stop words (such as the, of, them)

6. Lemminize (remove inflections, such as by changing plurals to singular)

This gets you to the point where you're ready to do linguistic analysis (which, as I say, I only assume is your goal).

Fortunately, there are packages, such as NLTK in Python that do this for you without the need for regex.

Writing a model to infer that from a training set of clean text might not be the best way to go, if getting to clean is your goal, rather than extending the frontiers of machine learning.

Could you post a MWE (Minimal Working Example) of a chunk of raw text and it's corresponding manually scrubbed version. It may turn out to be a quicker means to your end to apply a combination of off-the-shelf tools to get to where you want to be.