I have PDF text data that I want to use for NLP tasks. Now during preprocessing, I came to the conclusion that it is not worth analyzing the table of content. It merely generates a bunch of nonsensical tokens/sentences which are nothing but noise. Also, some of the PDFs include a list of abbreviations after the ToC, which I also want to remove.
The difficulty is to find a rule, which removes these parts but not remove anything from the body. I'm already removing lines of the document which contain less than 30 characters (it remains to be seen whether that's a good idea), but the ToC and the list of abbreviations are still there.
I'm using Python and I'm parsing the PDFs using TIKA, they don't contain images, however the layout is heterogeneous.
Is there a tool or a good practice to automatically remove tables of content and things like list of figures, list of abbreviations and so on from PDF documents parsed into python?