1
$\begingroup$

Let's say I want to create a Machine Learning system that has a lot of log files of some few types (F1, F2,.. Fn) and I get a new Log file with maybe some errors or missing data.

How do I classify it into these class types or classify it is an anomaly if it doesn't belong to anyone of them.

I thought about anomaly detection but couldn't figure how to parse structure information from the text classes like (F1, F2... .etc).

Also what kind of structural information to extract from text files?

These input classes contain 100 - 1000 lines of code per document of each class type.

I looked into Linting or DeepCode ...

A sample log file looks like this:

11-02-11 16:47:35,985 +0000 E Activity class {com.trackingeng/LandingActivity} does not exist.
12-02-11 17:47:35,985 +0000 I Starting: Intent { act=android.intent.action.MAIN
.....

A log file may have stack trace like this also:

Error:
    Error detail 1
    Error detail 2
    ....
Non-Error:
    .....
Warning:
    .....
and similar to this.

Any help in which direction to look for is greatly appreciated.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Please can you edit your post to provide an example of the input log files and their types. How many types in total do you expect to receive, or is it unbounded? $\endgroup$
    – BenP
    May 24, 2019 at 7:29
  • $\begingroup$ Just updated @BenP $\endgroup$
    – Kartikey Singh
    May 24, 2019 at 7:41
  • 1
    $\begingroup$ Thanks. Still hard to gauge without an example of log data from two or more separate classes. Is the content of each log of class n likely to be different? My initial thought is that you could build a 1-v-rest (or similar) multi-label classifier to determine the type of log based on typical content. $\endgroup$
    – BenP
    May 24, 2019 at 8:17
  • $\begingroup$ The data for each class is of similar manner but errors produced maybe from android, desktop .etc and the structure of errors i.e. tab spacing differentiates them.So, how to extract structural information from text to apply ML to it? $\endgroup$
    – Kartikey Singh
    May 24, 2019 at 9:00
  • $\begingroup$ So could you engineer features with regex to capture the: i) source OS, and ii) tabbed structural information in the error lines? For example the number of tabs, sequences of tabs, and use these to train a model? $\endgroup$
    – BenP
    May 24, 2019 at 9:20

1 Answer 1

Reset to default
0
$\begingroup$

Based on your current examples.

  1. You have an ML that is doing work across multiple systems and users. Generating logs.
  2. The logs can be grouped or classed (e.g. $F1, F2,...,FN$) by:
    • i) the operating System used (e.g. Android, Windows etc) and
    • ii) the error or message generated. The error messages are differentiated by tab white space.

If you cannot work out the log class using some logic, you could look to engineer features with regex to capture the source OS and the tabbed structural information in the error lines.

|---------------------|------------------|
| Error detail        |     n_tabs       |
|---------------------|------------------|
|  string_error       |         0        |
|---------------------|------------------|
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.