prepare email text for nlp (sentiment analysis)

Question

I have text of emails, which also contains disclaimers, phone numbers, email addresses, file attachment names, addresses, greetings etc.

At the moment I blindly pass this text through an OOTB sentiment analyser called Vader with poor results (i.e. if I open an email marked as negative, my human understanding does not confirm the sentiment - looking at the core English text).

I could use regex etc. to strip out email addresses, file names etc. but other text components (e.g. addresses, disclaimers) are harder to remove. Btw, disclaimers are often negative ...

Anyway, I wonder if anyone is aware of text prep methods for my scenario - to extract core human text. Google searches were moderately successful. Thanks!

Answer 1

I have done something similar in the past. I'll sketch an outline for you.

First you break the text into paragraphs and tokenize. Then write some regexp rules to capture the data you want to remove. For instance, if an email signature commonly contains a phone number, paragraph, and a website, you can count those features and flag it based on some threshold you decide.

Next, do likewise with the other features you mentioned. My experience is it's highly domain dependent so you really need to look at the data and use your best judgement.

The result of this process should be a data consisting of tokenized paragraphs where the paragraph has been labeled 'noise' or 'clean' based on the feature count.

From there, convert your token representation using tf-idf or another type of embedding. You should be able to use this as input to your favorite classifier, and I have had success using SVMs to that end.

The result is going to be biased towards your rules but you are also leveraging features that are in the labeled examples that are not explicitly in the rules. Particularly so for longer paragraphs.

It might seem a bit a bit janky but believe it or not it works.