How to extract features and classify alert emails coming from monitoring tools into proper category?

Question

My company provides managed services to a lot of its clients. Our customers typically uses following monitoring tools to monitor their servers/webapps:

1. OpsView
2. Nagios
3. Pingdom
4. Custom shell scripts

Whenever any issue is found, an alert mail comes to our Ops team so that they act upon rectifying the issue.

As we manage thousands of servers, our Ops teams' inbox is flooded with email alerts all the time. Even a single issue which has a cascading effect, can trigger 20-30 emails.

Now, what I want to do is to implement a system which will be able to extract important features out of an alert email - like server IP address, type of problem, severity of problem etc. and also classify the emails into proper category, like CPU-Load-Customer1-Server2, MySQL-Replication-Customer2-DBServer3 etc. We will then have a pre-defined set of debugging steps for each category, in order to help the Ops team to rectify the problem faster. Also, the feature extractor will provide input data to the team for a problem.

So far I have been able to train NaiveBayesClassifier with supervised learning techniques i.e. labeled training data(cluster data), and able to classify new unseen emails into its proper cluster/category. As the emails are based on certain templates, the accuracy of the classifier is very high. But we also get alert emails from custom scripts, which may not follow the templates. So, instead of doing supervised learning, I want to try out unsupervised learning for the same. I am looking into KMeans clustering. But again the problem is, we won't know the number of clusters beforehand. So, which algorithm will be best for this use case? Right now I am using Python's TextBlob library for classification.

Also, for feature extraction out of an alert email, I am looking into NLTK (http://www.nltk.org/book/ch07.html) library. I tried it out, but it seems to work on proper English paragraphs/texts well, however, for alert emails, it extracted a lot of unnecessary features. Is there already any existing solution for the same? If not, what will be the best way to implement the same? Which library, which algorithm?

PS: I am not a Data Scientist.

Sample emails:

PROBLEM: CRITICAL - Customer1_PROD - Customer1_PROD_SLAVE_DB_01 -  CPU Load Avg     Service: CPU Load Avg  Host: Customer1_PROD_SLAVE_DB_01  Alias: Customer1_PROD_SLAVE_DB_01  Address: 10.10.0.100  Host Group Hierarchy: Opsview > Customer1  - BIG C > Customer1_PROD  State: CRITICAL  Date & Time: Sat Oct 4 07:02:06 UTC 2014    Additional Information:     CRITICAL - load average: 41.46, 40.69, 37.91
RECOVERY: OK - Customer1_PROD - Customer1_PROD_SLAVE_DB_01 -  CPU Load Avg     Service: CPU Load Avg  Host: Customer1_PROD_SLAVE_DB_01  Alias: Customer1_PROD_SLAVE_DB_01  Address: 10.1.1.100  Host Group Hierarchy: Opsview > Customer1  - BIG C > Customer1_PROD  State: OK  Date & Time: Sat Oct 4 07:52:05 UTC 2014    Additional Information:     OK - load average: 0.36, 0.23, 4.83
PROBLEM: CRITICAL - Customer1_PROD - Customer1_PROD_SLAVE_DB_01 -  CPU Load Avg     Service: CPU Load Avg  Host: Customer1_PROD_SLAVE_DB_01  Alias: Customer1_PROD_SLAVE_DB_01  Address: 10.100.10.10  Host Group Hierarchy: Opsview > Customer1  - BIG C > Customer1_PROD  State: CRITICAL  Date & Time: Sat Oct 4 09:29:05 UTC 2014    Additional Information:     CRITICAL - load average: 29.59, 26.50, 18.49

Classifier code:(format of csv - email, <disk/cpu/memory/mysql>)

from textblob import TextBlob
from textblob.classifiers import NaiveBayesClassifier
import csv
train = []
with open('cpu.txt', 'r') as csvfile:
    reader = csv.reader(csvfile, delimiter=',', quotechar='"')
    for row in reader:
        tup = unicode(row[0], "ISO-8859-1"), row[1]
        train.append(tup)
// this can be done in a loop, but for the time being let it be
with open('memory.txt', 'r') as csvfile:
    reader = csv.reader(csvfile, delimiter=',', quotechar='"')
    for row in reader:
        tup = unicode(row[0], "ISO-8859-1"), row[1]
        train.append(tup)

with open('disk.txt', 'r') as csvfile:
    reader = csv.reader(csvfile, delimiter=',', quotechar='"')
    for row in reader:
        tup = unicode(row[0], "ISO-8859-1"), row[1]
        train.append(tup)

with open('mysql.txt', 'r') as csvfile:
    reader = csv.reader(csvfile, delimiter=',', quotechar='"')
    for row in reader:
        tup = unicode(row[0], "ISO-8859-1"), row[1]
        train.append(tup)

cl = NaiveBayesClassifier(train)
cl.classify(email)

Feature extractor code taken from: https://gist.github.com/shlomibabluki/5539628

Please let me know if any more information is required here.

Thanks in advance.

Answer 1

I want to try out unsupervised learning for the same. I am looking into KMeans clustering. But again the problem is, we won't know the number of clusters beforehand. So, which algorithm will be best for this use case?

When you don't know the number of clusters beforehand, it is still possible to do unsupervised learning using a Dirichlet process to sample parameters associated to clusters/groups and then cluster your tokens according to those parameters. The general idea is to use a Dirichlet distribution to generate probabilities over words for each cluster and a Dirichlet process uses these probabilities to assign a cluster to each word in your vocabulary. If you want to share clusters between emails, then you use Hierarchical Dirichlet Processes. Here you can find a nice blog post of how this works.

The most popular library for clustering is gensim, but notice their warning regarding the Hierarchical Dirichlet Process implementation:

gensim uses a fast, online implementation based on [3]. The HDP model is a new addition to gensim, and still rough around its academic edges – use with care.

As for feature extraction, your question doesn't say exactly what kind of unnecessary features you are getting, but if that's the case, you need to filter your tokens before or after processing them with NLTK. In general, you can't expect excellent results for very specific applications.