2
$\begingroup$

I currently have a large stream of data, with points such as HTTP request/response codes (200, 404, 500, etc.). Essentially, I want to perform anomaly detection for when too many signals that are NOT 200 are received. This means that the signal to be analyzed is dependent on groupings of data points (i.e., a single data point indicating that a 404 was sent is not good enough, only a cluster of 404's close together with respect to time means something).

Is there a good algorithm/methodology to approach this? I was previously thinking of having a moving window counter of the previous codes and basing it off of this, but I feel like the window size for example, is very subjective, and I am not sure how to tune that.

$\endgroup$
1
$\begingroup$

Look at the rate instead of single requests.

There is probably a "usual" frequency of non-success codes. If you observe an unusual frequency, you have an event.

If you want a simple algorithm: Consider that you want to raise an alarm if you observe a rate much higher than, say 5%.

To add some margin, will increase this to 0.1, and a threshold of about 10 errors "too much".

Use a counter x, initially 0, updated using

$$ x_{t+1} \leftarrow\begin{cases} \max(0, x_t - 0.1) & \text{if request successful} \\ x_t + 1 & \text{if request causes an error} \end{cases} $$ then alert if this value reaches 10.

$\endgroup$
1
$\begingroup$

It seems that your solution does not have to include an "anomaly detection" in context of data analysis, but can more easily be solved with an iteration through the vector assuming a chronological order of the data (you did not mention any other feature so the problem is 1D).

I would suggest the following approaches:

1) Iterate through the vector and if you find N > threshold non-2xx responses in a row, mark starting index of a "cluster".

    vector<int> status, indices;
    int threshold = 10, counter = 0, flag = 0;
    status = read_data("data.csv");
    for(int i = 0; i < status.size(); i++) {
            if (status[i] < 200 && status[i] > 299) {
                    if (++counter >= threshold && !flag) {
                            indices.push_back(i - counter);
                            flag = 1;
                    }
            } else {
                    counter = 0;
                    flag = 0;
            }
    }
    return indices;

2) Once you cross over some non-2xx example, initialize the weight to, let's say 0.0 (zero). For every new negative example, subtract, let's say 0.2 and for a positive example add 0.05 or something and if you end up bellow -2.0 or above 2.0, reset the procedure and in case that it's < -2.0, return the starting index. You can play around with given numbers.

3) Some more complex strategy?

But yes, your solution does not have to include a machinery, try to start from something more simple and it will lead you in the right direction.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.