My company provides managed services to a lot of its clients. Our customers typically uses following monitoring tools to monitor their servers/webapps:
- OpsView
- Nagios
- Pingdom
- Custom shell scripts
Whenever any issue is found, an alert mail comes to our Ops team so that they act upon rectifying the issue.
As we manage thousands of servers, our Ops teams' inbox is flooded with email alerts all the time. Even a single issue which has a cascading effect, can trigger 20-30 emails.
Now, what I want to do is to implement a system which will be able to extract important features out of an alert email - like server IP address, type of problem, severity of problem etc. and also classify the emails into proper category, like CPU-Load-Customer1-Server2, MySQL-Replication-Customer2-DBServer3
etc. We will then have a pre-defined set of debugging steps for each category, in order to help the Ops team to rectify the problem faster. Also, the feature extractor will provide input data to the team for a problem.
So far I have been able to train NaiveBayesClassifier with supervised learning techniques i.e. labeled training data(cluster data), and able to classify new unseen emails into its proper cluster/category. As the emails are based on certain templates, the accuracy of the classifier is very high. But we also get alert emails from custom scripts, which may not follow the templates. So, instead of doing supervised learning, I want to try out unsupervised learning for the same. I am looking into KMeans clustering. But again the problem is, we won't know the number of clusters beforehand. So, which algorithm will be best for this use case? Right now I am using Python's TextBlob library for classification.
Also, for feature extraction out of an alert email, I am looking into NLTK (http://www.nltk.org/book/ch07.html) library. I tried it out, but it seems to work on proper English paragraphs/texts well, however, for alert emails, it extracted a lot of unnecessary features. Is there already any existing solution for the same? If not, what will be the best way to implement the same? Which library, which algorithm?
PS: I am not a Data Scientist.
Sample emails:
PROBLEM: CRITICAL - Customer1_PROD - Customer1_PROD_SLAVE_DB_01 - CPU Load Avg Service: CPU Load Avg Host: Customer1_PROD_SLAVE_DB_01 Alias: Customer1_PROD_SLAVE_DB_01 Address: 10.10.0.100 Host Group Hierarchy: Opsview > Customer1 - BIG C > Customer1_PROD State: CRITICAL Date & Time: Sat Oct 4 07:02:06 UTC 2014 Additional Information: CRITICAL - load average: 41.46, 40.69, 37.91
RECOVERY: OK - Customer1_PROD - Customer1_PROD_SLAVE_DB_01 - CPU Load Avg Service: CPU Load Avg Host: Customer1_PROD_SLAVE_DB_01 Alias: Customer1_PROD_SLAVE_DB_01 Address: 10.1.1.100 Host Group Hierarchy: Opsview > Customer1 - BIG C > Customer1_PROD State: OK Date & Time: Sat Oct 4 07:52:05 UTC 2014 Additional Information: OK - load average: 0.36, 0.23, 4.83
PROBLEM: CRITICAL - Customer1_PROD - Customer1_PROD_SLAVE_DB_01 - CPU Load Avg Service: CPU Load Avg Host: Customer1_PROD_SLAVE_DB_01 Alias: Customer1_PROD_SLAVE_DB_01 Address: 10.100.10.10 Host Group Hierarchy: Opsview > Customer1 - BIG C > Customer1_PROD State: CRITICAL Date & Time: Sat Oct 4 09:29:05 UTC 2014 Additional Information: CRITICAL - load average: 29.59, 26.50, 18.49
Classifier code:(format of csv - email, <disk/cpu/memory/mysql>)
from textblob import TextBlob
from textblob.classifiers import NaiveBayesClassifier
import csv
train = []
with open('cpu.txt', 'r') as csvfile:
reader = csv.reader(csvfile, delimiter=',', quotechar='"')
for row in reader:
tup = unicode(row[0], "ISO-8859-1"), row[1]
train.append(tup)
// this can be done in a loop, but for the time being let it be
with open('memory.txt', 'r') as csvfile:
reader = csv.reader(csvfile, delimiter=',', quotechar='"')
for row in reader:
tup = unicode(row[0], "ISO-8859-1"), row[1]
train.append(tup)
with open('disk.txt', 'r') as csvfile:
reader = csv.reader(csvfile, delimiter=',', quotechar='"')
for row in reader:
tup = unicode(row[0], "ISO-8859-1"), row[1]
train.append(tup)
with open('mysql.txt', 'r') as csvfile:
reader = csv.reader(csvfile, delimiter=',', quotechar='"')
for row in reader:
tup = unicode(row[0], "ISO-8859-1"), row[1]
train.append(tup)
cl = NaiveBayesClassifier(train)
cl.classify(email)
Feature extractor code taken from: https://gist.github.com/shlomibabluki/5539628
Please let me know if any more information is required here.
Thanks in advance.