How to detect when the “bibliography” of a paper has began?

Question

I'm working on project that involves reading papers from ArXiV to look for particular patterns (It gets complicated but it basically has to do with common phrases and expressions) in their text.

My process can be abstractly described as:

$$ PDF \rightarrow "\text{PDF as a string}" \rightarrow \text{Pattern_Finding_ Method}("\text{PDF as a string}") $$

Now unfortunately, the "bibliography" or references section of many papers cause the Pattern_Finding_Method to break, because they just happen to carry a lot of the behavior we are looking for (but they are not of interest).

As a human it's quite easy for me just manually read and see where the "references" begin and then I can trim that part of the pdf string but at scale this is not practical. Moreover paper writers don't have consistent ways of declaring when their "references" "bibliography" or "acknowledgements" begin.

So it seems natural to view this as an ML/AI problem, where I have a string, I have a loose notion of what constitute the "references" of the string which I can provide training data for (I have the pdf as a string and I can list a character index on the string the references begin)

Now given training data I need to come up with some kind of model that can effectively learn how to detect references on its own.

This is where I get stuck. The data problem i'm dealing with is a highly semantic problem (it's particular organizations of words and their underlying meaning and patterns to how those words are displayed that is giving me cues as to when the reference have began), but my knowledge of learning algorithms is limited to mostly geometric data (SVMs), or at least highly continuous data (Neural Network Models), and then in the case of NLP, my understanding is at best a recipe book of very goal specific algorithms: (ex: TF-IDF for document classification).

I don't know how to bridge the gap from my understanding to creating a specialized model for the problem at hand; that I have intuitive reason to believe will work.

Formal Problem Statement:

Given a collection of large strings, each attached with an integer $i$ indicating where the references begin, determine a model that can reliably detect when references begin on new text.

Answer 1

Here are a few heuristics that could help solve the problem without ML:

• If all the papers are from the Arxiv, you can download the source tex files and regenerate the paper without the bibliography

• You can use Arxiv-Vanity or their code to render the document into HTML (or some intermediate representation), then use that hierarchical structure to easily remove the part with a bibliography

• Look for the word "references" followed by high word density of the words "in", "pages" and years in the 1900s and early 2000s

I guess what I am saying is "the input already has a fairly rigid structure, why don't you use that instead of statistics?"