Understanding of spikes and comparing different time-series data

Question

Complete newbie here so, do forgive any misunderstandings that I may have.

Currently, we have a lot of metrics to track each detail of our backend applications like traffic, response times, memory load, upstream response times, etc using a time-series solution (Prometheus) and have dashboards similar to what MediaWiki has

When an issue happens like a sudden spike in response times, we go over the graphs and try to manually figure out the issue from the graphs. The steps to debug are standard which can be represented easily with a flowchart

I would like to automate these manual steps and as I understand, I need to build a basic understanding of Data Science for things like:

• Understanding whether a spike has occurred
• Comparing the trends between different time-series data to be able to conclude if one is the cause of the other.
• Even if the solution can't find the right match but it would still be a win if it can narrow down the areas for users to look into.
• Ignoring minor fluctuations to understand the overall trend

For this, I have the following questions:

1. What are the areas or concepts that I should be familiarizing myself with?
2. What existing solutions or building blocks like modules I can use for my use case?

To add, I am not looking for a forecast here and the data will be continuous.

Answer 1

This is commonly time-series anomaly detection which is a complex field of study.

Understanding whether a spike has occurred or not typically starts with hard-coded thresholding and later, learned patterns. Common techniques to learn patterns include STL (seasonal-trend decomposition), Isolation Forest, and time series clustering.

Granger causality is one way to compare different time-series data to see if one time series is useful in forecasting another.

Answer 2

You can first learn the time series properties by modeling them using a method which determines the Trend, Seasonality, and the Intercept. One such algorithm is the Holt-Winters algorithm. E.g.

from statsmodels.tsa.holtwinters import ExponentialSmoothing

esm = ExponentialSmoothing(ts).fit()  # ts is the time series

# Then retrieve trend and other components from the `esm` model

On the same model, you now need to build the upper and lower bounds for your data. You can do so by leveraging the residual errors and other components. I can expand on that if needed.

Answer 3

Simple spike detection (what I would do)

• Get your time series data into a pandas dataframe in python
• iterate over each element, and see how much greater it is than the, lets say, 2 hour average. Let's call that spikeness
• if spikeness(i) > threshold, it's a peak
• you'll get something like this, 00000000011111111000000111111110000, finding when it flips between 0 and 1 is a good solution. You might need to "debounce" this signal.

more advanced spike detection

check out sklearn

comparing trends

Data science isn't a free lunch, and the most powerful neural network is your own head. I'm skeptical, for non value added work that is as volatile as bug tracking, you'll be able to get results that justify the cost. However:

• looking into clustering seems reasonable. This might help you see the data differently, and isn't too hard to set up (t-sne is a good one)
• you could train a time series LSTM or CNN to try to learn. I don't recommend this approach for your use case

the rub

You're stepping into an incredibly deep pond, but it seems like your pebble is right on the edge. You could dive in, but I would recommend really analyzing what you need and trying to implement the simplest solution to meet that goal. Setting thresholds compared to the baseline (2x the average over the last day, for instance) for each graph seems like a great place to start.

Some related things in this deep pond, that I don't think you need to spend copious amounts of time on:

• clustering
• supervised learning
• time series analysis
• CNN
• LSTM
• signal processing
• frequency domain analysis
• sklearn
• pandas
• numpy

I think data science suffers from glist and glam, with people wanting to over engineer solutions to problems that aren't even really problems. The first step of a ML/data science project is bringing those pie in the sky ideas back down to reality.

Answer 4

interesting a practical problem. As suggested by others below, you may be interested in "time series anomaly detection" and "causality". So you already got a response to your 1st answer. However, you'll soon realize that in the literature "anomaly detection" is often an UNsupervised task (meaning that yes, you can spot anomalies, but if you are not really aware of how the underlying system works it's difficult to define if the "weird" condition is strictly an anomaly).

Let me try providing some food for thought:

1. Understanding whether a spike has occurred: I guess you have a practical feeling of acceptable "Level Of Service" for each metric. For instance, your machines could be sized so that you get an average response time of 50ms. A second step would be getting the distribution of the response time for each given hour, i.e. you may want to serve 95% of users in 35ms. That could be an example of simple statistics to build monitoring. Warlax called this "spikiness", but what it's worth stressing is that it should come from your level of service.

2. Comparing the trends between different time-series data to be able to conclude if one is the cause of the other. A general solution to this kind of problem is not straightforward. Once again it may be useful to think about engineering considerations: if you get a ton of pages to serve => CPU load increases (and that's ok). The other way round may be untrue: if CPU load increases in low traffic conditions you may just have a bug. Documenting these simple rules may be an excellent starting point for consultants or for the evaluation of more sophisticated methods.

3. Even if the solution can't find the right match but it would still be a win if it can narrow down the areas for users to look into. A good starter is the "Level Of Service" mentioned in (1). Another keyword to search could be "KPI". E.g. think of defining 3 indicators (avg response time in the last hour, avg CPU load, avg failed requests): if they are systematically high you might place a red background to some part of your time histories.

4. Ignoring minor fluctuations to understand the overall trend. Iterating the process above you may learn if your statistics are good enough to filter out "false alarms" and guarantee a good service or if you really need to design and engineer "Data Science" solutions.