How to identify and extract patterns from emails

Question

I would like to know if it would be possible to identify some patterns in a text. For example, looking at emails, there is some common words used at the beginning and at the end.

Dear Mr Pascal, 

We regret to inform you that we will not be able to respect the deadline previously agreed for the delivery of your order. Our supplier has warned us today that they are experiencing supply problems, which will result in a delay in our production chain. We count on your understanding and thank you for your patience. 

Please accept our apologies.

Best regards,

Matt

If I look at emails, they usually start with Dear/Hi/Hello/Good morning..., then the title/name of a person/company; the body; and the conclusion (I look forward to hearing from you; kind regards, best regards;....). I would like to ask you, therefore, if there is a way to collect information about these patterns and also if it is possible to classify emails by the patterns.

Answer 1

It depends exactly on which kind of patterns you are talking about. Are they deterministic? That is, they are all the same, so you want to get everything after Dear, or before Att / Best Regards, you can explore regular expression patterns. In python, you can use re library:

https://docs.python.org/3/library/re.html

There are books about regular expressions, so this is a big topic and a strong start if you want to check patterns.

Other kind of tasks you could do:

• Word Embeddings on raw e-mails to check word similarity.
• Clustering e-mails using Bag-of-Words or Word Embeddings.
• Label af set o e-mails and train a supervised algorithm to predict an outcome of interest.

Anyway, text mining is really a big discipline, so if you want a more precise answer, I would recommend to narrow down your question.

Hope it helps.

Answer 2

It depends on which kind of task you want to perform at the end. From what I understood from your question, you have emails with same pattern occurring in the beginning as well as at the end. You want to perform a classification tasks on the emails based on the real sense of email excluding subject and conclusion. There are multiple ways you can do this, for example as follows:

• You can use Active Learning in order to classify the emails based on little human intervention for each batch or iteration. Before that you will have to first properly preprocess the emails using any regex library. Active learning need some of user input in order to control the model understanding for each batch of emails.
• You can also use an unsupervised learning methodology like Topic Modeling (LDA, LSA) which yields an cluster of emails with patterns detected by the model as topic. The topic is full proof or evidence for model to cluster those email. I think this is what you actually need. Topic model.
• You can use a Bidirectional context extraction model like BERT or ELMO to get document representation and later you would map to its' label.

I hope this explanation will help to you.