1
$\begingroup$

I have a list of email subjects like

<XYZ> commented on <ABC>
Weekly review for <Company>
Your account is ready 

And I want to find patterns in them so I can group them.

Is there a well known algorithm I can use?

$\endgroup$
  • $\begingroup$ Unsupervised for the most part. Little input can be given by the user as a correction, but not during the calculation if that makes sense. The number of different emails is not known. $\endgroup$ – hakunin Mar 6 '16 at 14:06
  • 1
    $\begingroup$ @RicardoCruz Seems reasonably on-topic to me... It's kinda broad and kinda vague but requests for algorithms look like computer science to me. $\endgroup$ – David Richerby Mar 6 '16 at 17:30
  • $\begingroup$ Requests for software implementations, tools, or libraries are off-topic here, so I've edited that part out of your question. Requests for algorithms or techniques are on-topic here, so with this edit, the question seems on-topic to me. (Cc: @RicardoCruz) Note to hakunin: please don't leave clarifications in the comments. Instead, edit your question to add the missing information. We want questions to be self-contained: Readers shouldn't have to read the comments to understand your question. $\endgroup$ – D.W. Mar 6 '16 at 18:55
  • $\begingroup$ @RicardoCruz, in the future, if you're going to mention another site, please remind the person not to cross-post on multiple SE sites, and tell them how they can migrate their question. That violates SE rules, so by suggesting another site, you're basically encouraging the poster to do something that will get them in trouble, which isn't a great user experience for them. I know this isn't intuitive/obvious, so I thought I'd mention it for your future consideration. Thank you! $\endgroup$ – D.W. Mar 6 '16 at 18:57
  • $\begingroup$ (@D.W. Duly noted!) Poster: If you want something fast, I think something you may want to try is hierarchical clustering (because you don't need to predefine the number of clusters, you can prune it later), together with a text mining measure such as something simple like TF-IDF cosine similarity. For something more elaborate, I would use something based on word embeddings. $\endgroup$ – Ricardo Cruz Mar 7 '16 at 17:56
3
$\begingroup$

You'll probably have to experiment a bit with different approaches. Let me outline two different kinds of approaches you could try.

Topic modelling

You could try applying unsupervised topic modelling to your subject lines. LDA is probably the most widely used method.

Topic modelling tries to find a limited number of "topics", and assigns each subject line to one or more "topics" based on the words in the subject line.

Clustering

You could try using clustering. Broadly, you would find some way map each subject line to a feature vector, and then apply some unsupervised clustering method. There are many options for each of those two steps. To get feature vectors, you might try any of the existing word embeddings; e.g., you might try word2vec. For clustering, there are many, many clustering algorithms; e.g., you might try k-means. I recommend you do a little reading on these topics and then experiment with them a bit.

Caveats

A warning. Don't set your expectations too high. Subject lines are typically very short, and that will make it hard for these techniques to find clusterings. In other words, you are operating in a regime that's known to be hard for existing NLP and ML techniques.

If you can find any other features that might assist with clustering similar emails (e.g., the identify of the sender? the mailing list it was sent on, if any?), including that information might improve the quality of the clusters.

| improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.