I have a corpus of text files (free books) which are poorly formatted. The goal is to extract a particular chapter (say chapter 2) from the raw text with all weird formatting removed. Some documents are literal copy and pastes from pdfs, so that the text contents from a chapter would have page numbers interspersed throughout, etc., and I.e., it would be impossible to create a good regex rule.
I've manually gone through a number of documents and stripped out all unwanted formatting to arrive at the desired, clean section of text. Using this as the training set, what would be a good model to set up for this task?
To clarify, the input would be a (very long) string, and the output would also be a string.
EDIT:
Please note that this particular example can be very easily handled by a series of regex rules, but it's only for illustration purposes only. Imagine if each document had their own minor idiosyncrasies making it very difficult to come up with a general series of regex rules to apply to all docs in the corpus.
Sample input:
Book Title ABC
Table of Contents
Preface ................................ Pg 1
Chapter 1 .............................. Pg 2
Chapter 2 .............................. Pg 3
Chapter 3 .............................. Pg 4
Preface
<p> Contents for the preface sit here. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse scelerisque turpis ac commodo porttitor. Suspendisse iaculis lacinia ante, non suscipit purus venenatis et. </p>
- 1 -
Leftover Contents from Preface
Chapter 1
Praesent at justo sit amet nulla porttitor hendrerit. Etiam pretium sem ac feugiat posuere. Nulla mi risus, mollis sit amet ligula egestas, lacinia dapibus tellus. Proin iaculis ligula ultricies nunc elementum maximus. Nulla accumsan libero at nunc scelerisque, ut fermentum lacus euismod. Nulla imperdiet pharetra laoreet. Quisque id mollis diam. Vestibulum interdum orci at lectus ullamcorper pharetra. Curabitur laoreet mollis pharetra.
- 2 -
Chapter 2
Cras diam nibh, congue sit amet imperdiet quis, finibus imperdiet tellus. Sed imperdiet risus id elit consequat cursus. Curabitur consequat facilisis molestie. Ut varius urna vel ornare scelerisque. Nullam dictum tempus sapien, in pulvinar tellus consectetur non. Vestibulum consequat pretium iaculis.
- 3 -
Leftover contents from Chapter 2. Vivamus pretium mauris at metus egestas, eget luctus ligula auctor.
Chapter 3
Maecenas vel dapibus lorem. Sed eget justo sit amet libero aliquam maximus a quis augue. Nunc consequat, urna quis elementum condimentum, quam arcu consectetur tellus, vehicula gravida est purus id nisl.
- 4 -
Nulla eget molestie velit, pharetra euismod magna. Vestibulum ultricies justo vitae massa vulputate, et venenatis orci fringilla.
Index
Sample Output:
Cras diam nibh, congue sit amet imperdiet quis, finibus imperdiet tellus. Sed imperdiet risus id elit consequat cursus. Curabitur consequat facilisis molestie. Ut varius urna vel ornare scelerisque. Nullam dictum tempus sapien, in pulvinar tellus consectetur non. Vestibulum consequat pretium iaculis.
Leftover contents from Chapter 2. Vivamus pretium mauris at metus egestas, eget luctus ligula auctor.