I have a corpus of unstructured text that, due to a concatenation from different sources, has boilerplate metadata that I would like to remove. For example:
DESCRIPTION PROVIDED BY AUTHOR: The goal of my ...
Author provided: The goal of my ...
The goal of my ... END OF TRANSCRIPT
The goal of my ... END, SPONSORED BY COMPANY XYZ
The goal of my ... SPONSORED: COMPANY XYZ, All rights reserved, date: 10/21
This boilerplate can be assumed to occur in beginning or end of each sample. What are some robust methods for wrangling this out of the data?