I have text of emails, which also contains disclaimers, phone numbers, email addresses, file attachment names, addresses, greetings etc.

At the moment I blindly pass this text through an OOTB sentiment analyser called Vader with poor results (i.e. if I open an email marked as negative, my human understanding does not confirm the sentiment - looking at the core English text).

I could use regex etc. to strip out email addresses, file names etc. but other text components (e.g. addresses, disclaimers) are harder to remove. Btw, disclaimers are often negative ...

Anyway, I wonder if anyone is aware of text prep methods for my scenario - to extract core human text. Google searches were moderately successful. Thanks!


I have done something similar in the past. I'll sketch an outline for you.

First you break the text into paragraphs and tokenize. Then write some regexp rules to capture the data you want to remove. For instance, if an email signature commonly contains a phone number, paragraph, and a website, you can count those features and flag it based on some threshold you decide.

Next, do likewise with the other features you mentioned. My experience is it's highly domain dependent so you really need to look at the data and use your best judgement.

The result of this process should be a data consisting of tokenized paragraphs where the paragraph has been labeled 'noise' or 'clean' based on the feature count.

From there, convert your token representation using tf-idf or another type of embedding. You should be able to use this as input to your favorite classifier, and I have had success using SVMs to that end.

The result is going to be biased towards your rules but you are also leveraging features that are in the labeled examples that are not explicitly in the rules. Particularly so for longer paragraphs.

It might seem a bit a bit janky but believe it or not it works.

  • $\begingroup$ thanks just curious how do you identify paragraphs rather than sentences? $\endgroup$
    – cs0815
    Feb 7 '19 at 16:16
  • 1
    $\begingroup$ Depends on the input data format. If you read it line by line you can look for a blank line and separate them that way. Otherwise if it's all in a giant string you can look for the newline \n and split them on that. $\endgroup$
    – Sledge
    Feb 7 '19 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.