Anomaly detection in a database

Question

We have a production database. The load on the database varies at different times. I want to identify anomalies; for e.g, the number of database processes responding to user queries at 9 am is 100 for a given day. If the number is 200, then it's an anomaly and as the DBA, we need to check the DB immediately. The goal is to identify a pattern and alert when there's an event outside this pattern.

Day  time  processcount label
Mon  09:00 100          Normal
Mon  09:05 150          Normal
Tue  09:00 200          Abnormal

I'm using pandas to collect the data but I'm not sure how to identify the pattern and report anomalies. The closest i could get is this thread How do I approach grouped anomaly detection?

Answer 1

There are some packages for anomaly detection such as Linkedin's Luminol (https://github.com/linkedin/luminol) or Microsoft's TagAnomaly (https://github.com/microsoft/TagAnomaly). Also you can use clustering algorithms for your features and detect outlier clusters. Or you can tag your previous anomalies to train your data (however its not possible most of the cases)

So my suggestion is prepare your data with features such that 5_min_window_processcount_mean, 5_min_window_processcount_std, 5_min_window_processcount_std and 5_min_lag_processcount_std etc.(get lagged and windowed features and then calculate their mean,std,median etc)

After that try clustering and check it if it can find anomalies or not. If not try to label your data with those features and try classification algorithms. Meantime you can use above packages with your features. (time based lag/window features are very important).