# Detecting boilerplate in text samples

I have a corpus of unstructured text that, due to a concatenation from different sources, has boilerplate metadata that I would like to remove. For example:

DESCRIPTION PROVIDED BY AUTHOR: The goal of my ...

Author provided: The goal of my ...

The goal of my ... END OF TRANSCRIPT

The goal of my ... END, SPONSORED BY COMPANY XYZ

This boilerplate can be assumed to occur in beginning or end of each sample. What are some robust methods for wrangling this out of the data?

• IMHO data-wrangling should be a tag here, but I lack the reputation to create it. – Hooked Nov 30 '15 at 17:45

This might get you started. Phrase length is determined by the range() function. Basically this tokenizes and creates n-grams. Then it counts each token. Tokens with a high mean over all documents (occurs often across documents) is printed out in the last line.

from sklearn.feature_extraction.text import CountVectorizer
import numpy as np
import nltk

text = """DESCRIPTION PROVIDED BY AUTHOR: The goal of my a...
Author provided: The goal of my b...
The goal of my c... END OF TRANSCRIPT
The goal of my d... END SPONSORED BY COMPANY XYZ
"""

def todocuments(lines):
for line in lines:
words = line.lower().split(' ')
doc = ""
for n in range(3, 6):
ts = nltk.ngrams(words, n)
for t in ts: doc = doc + " " + str.join('_', t)
yield doc

cv = CountVectorizer(min_df=.5)

fit = cv.fit_transform(todocuments(text.splitlines()))
vocab_idx = {b: a for a, b in cv.vocabulary_.items()}

means = fit.mean(axis=0)
arr = np.squeeze(np.asarray(means))
[vocab_idx[idx] for idx in np.where(arr > .95)[0]]
# ['goal_of_my', 'the_goal_of', 'the_goal_of_my']

• tf-idf is a good idea, but it would require me to know beforehand the size of the boilerplate text. I guess I could keep taking steps up and threshold them, – Hooked Dec 1 '15 at 21:09
• This uses counts, not tf-idf. The idf would diminish a token value if it appeared in many documents. Also if you had a frequent text 'a b c d e' where each letter represents a word and you only decided to have a maximum of 3-word phrases then a_b_c, b_c_d, and c_d_e would also be frequent. So, it may not be necessary to know the maximum length of frequent phrases in advance. – user13684 Dec 1 '15 at 21:22