10
$\begingroup$

I'm trying to build a data set on several log files of one of our products.

The different log files have their own layout and own content; I successfully grouped them together, only one step remaining...

Indeed, the log "messages" are the best information. I don't have the comprehensive list of all those messages, and it's a bad idea to hard code based on those because that list can change every day.

What I would like to do is to separate the indentification text from the value text (for example: "Loaded file XXX" becomes (identification: "Loaded file", value: "XXX")). Unfortunately, this example is simple, and in real world there are different layouts and sometimes multiple values.

I was thinking about using string kernels, but it is intended for clustering ... and cluseting is not applicable here (I don't know the number of different types of messages and eventhough, it would be too much).

Do you have any idea?

Thanks for your help.

P.S: For those who programs, this can be easier to understand. Let's say that the code contains as logs printf("blabla %s", "xxx") -> I would like to have "blabla" and "xxx" seperatated

Improve this question
$\endgroup$
2
  • $\begingroup$ Can you provide a representative selection of examples that demonstrate the variety of items the algorithm will need to parse? $\endgroup$
    – Emre
    Commented Nov 21, 2014 at 4:20
  • 2
    $\begingroup$ There are a hundred ways to do this. Give some sense of what tools or language you need to do this in. Is there a data science aspect to this? seems like just log parsing. $\endgroup$
    – Sean Owen
    Commented Nov 21, 2014 at 10:32

3 Answers 3

Reset to default
3
$\begingroup$

How about considering each string as a process trace and applying alpha-algorithm? That would give you a graph and nodes with a big number out-edges will most likely point to values.

You can mark these nodes and for every new string parse/traverse the graph until you reach those areas.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks a lot. I didn't knew alpha-algorithms. I'll check in that direction. $\endgroup$
    – Michael Hooreman
    Commented Dec 15, 2014 at 12:17
2
$\begingroup$

This does not seem a Data Science problem. However there are very nice tools to do exactly that, checkout: logstash, flume and fluentd. Actually if you want to be able to filter in fast and "smart" way checkout Kibana from the guys of ElastichSearch (http://www.elasticsearch.org/overview/kibana). Those tools are enough to solve your problem in a very efficient way.

Improve this answer
$\endgroup$
1
  • $\begingroup$ My though is that feature extraction is part of data science. Well, it's just an opinion ;-) More seriouly, I'm not talking about easy standard log files, but custom ones from a very specific software. So, that's really extracting information from context « sentences » $\endgroup$
    – Michael Hooreman
    Commented Jun 6, 2019 at 19:17
1
$\begingroup$

If you're simply trying to separate textual and numeric information then there is a solution based on regular expressions or even just string splitting.

You could even do something like finding the first numeric character and split the text in half right before that.

With regular expressions you can match all numeric characters that follow eachother. The pattern would be ([0-9]+) with a global flag. It would match all the groups of numbers and you can do whatever you with with them afterwards.

Regex Tester is good for playing around with that stuff.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks Laurik. Unfortunately not only numbers, and I don't know also what future messages will be. So, I really need AI. $\endgroup$
    – Michael Hooreman
    Commented Dec 10, 2014 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.