1
$\begingroup$

I am currently working on parsed log data from IoT devices (log entries are batched every 15 minutes). All info_* / err_* are counts of the related log events:

| machine_id | timestamp           | os_ver | days_since_reboot | cpu_load | info_featA | info_featB | err_badrequests | err_timeoutrequests |
|------------|---------------------|--------|-------------------|----------|------------|------------|-----------------|---------------------|
| 1          | 2020-09-18 12:25:00 | 0.01   | 12                | 0.59     | 12         | 543        | 0               | 0                   |
| 1          | 2020-09-18 12:40:00 | 0.01   | 12                | 0.83     | 1          | 23         | 1               | 0                   |
| ...        | ...                 | ...    | ...               | ...      | ...        | ...        | ...             | ...                 |
| 2          | 2020-10-28 11:40:00 | 0.02   | 2                 | 0.12     | 43         | 54         | 0               | 1                   |
| 2          | 2020-10-28 11:55:00 | 0.02   | 2                 | 0.43     | 32         | 54         | 3               | 5                   |
| 2          | 2020-10-28 12:10:00 | 0.02   | 2                 | 0.23     | 0          | 4          | 0               | 23                  |

I am tasked to find some common patterns that might lead to an error event (e.g an increase in info_featA entries precedes a positive count in err_badrequests). For this reason I've thought to re-encode error columns into a boolean (True if err_* > 0) but I am fairly lost on the right approach to go about sequence analysis or finding the conditions that can lead to an error.

Also, error type is not very important for my problem as I just care about the presence of a positive error count (regardless of its type):

| machine_id | timestamp           | os_ver | days_since_reboot | cpu_load | info_featA | info_featB | has_err |
|------------|---------------------|--------|-------------------|----------|------------|------------|---------|
| 1          | 2020-09-18 12:25:00 | 0.01   | 12                | 0.59     | 12         | 543        | 0       |
| 1          | 2020-09-18 12:40:00 | 0.01   | 12                | 0.83     | 1          | 23         | 1       |
| ...        | ...                 | ...    | ...               | ...      | ...        | ...        | ...     |
| 2          | 2020-10-28 11:40:00 | 0.02   | 2                 | 0.12     | 43         | 54         | 0       |
| 2          | 2020-10-28 11:55:00 | 0.02   | 2                 | 0.43     | 32         | 54         | 1       |
| 2          | 2020-10-28 12:10:00 | 0.02   | 2                 | 0.23     | 0          | 4          | 0       |

At the moment I have created lagged variables (up to 2 hours before) to check if any lagged variable is somewhat correlated with error counts through a correlation matrix. I am pretty sure this is not the way to approach such problem.


The main goal is to identify (if any) some common patterns in the observed metrics that lead to an error.

$\endgroup$

1 Answer 1

1
$\begingroup$

There are several different ways to model this.

One option is to simplify the problem ignoring time. Fit a machine learning model that find the features that are associated with presence of errors.

Another option is treat this as event-prediction in time series data. In this case, it would be multivariate time series.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.