I have a set of unstructured data consisting command output logs for different operating systems like Unix, Windows, etc.

For example:

Releasing version 0.0.1 for Stackoverflow, on 01/01/2019. The coverage is 99% and build is passed.

(This is just an example, and not related to actual use case). This output is different for different operating systems. I want to perform tagging on this data

For the case above, the output should be:

Releasing version VERSION_NUMBER for PRODUCT_NAME, on releaseDate. The TEST_TYPE is TEST_VALUE and TEST_TYPE is TEST_VALUE

Here, some words are replaced with their corresponding sample tags.

I have studied techniques like POS tagging, NER, LSTM, but I don't know which one is suitable for this particular problem. How can I gather data from raw output and how to apply those techniques here.

Thanks to everyone who is willing to help me with this.


1 Answer 1


PoS tagging works for natural language only and identifies grammatical parts of the sentence, nothing more. LSTM is an algorithm that can be used to predict series. Named Entity Recognition (NER) and Terminology Extraction could work if you have already data to engage in Information Extraction (IE). However, In order to use a these techniques you need to have a trained model and in order to train one you need data. In your case that would involve identifying and tagging parts of the sentence by hand and later training a model with that data.

The best approach, in my opinion, is just use regex to identify parts of the sentence as one of the approaches of Information Extraction and use hard coded rules to best identify what you are trying to replace later.

For instance, if you want to search the version in the example:

import re

s = 'Releasing version. 0.0.1 for Stackoverflow, on 01/01/2019. The coverage is 99% and build is passed.'

re.search(r'Releasing version(.*?)for', s).group(1)

' 0.0.1 '

Check out these resources that will help you parse a log in Python using Regex.



But if you prefer to use Named Entity Recognition or Terminology Extraction techniques you could hack a NER model and train it yourself with your data. Keep in mind though, that according to Poibeau, Thierry; Kosseim, Leila (2001). "Proper Name Extraction from Non-Journalistic Texts". Language and Computers. 37 (1): 144–157.:

Research indicates that even state-of-the-art NER systems are brittle, meaning that NER systems developed for one domain do not typically perform well on other domains.

And according to Wikipedia: Considerable effort is involved in tuning NER systems to perform well in a new domain; this is true for both rule-based and trainable statistical systems.

So even if you pull that off, accuracy will be less than if you just extracted data using regex. An approximate of a reduction of 97% to 93% only for named entities (companies, names, etc.) Accuracy reduction will be much less in you case.

Check this link for more information about Information Extraction: https://web.stanford.edu/~jurafsky/slp3/17.pdf

  • $\begingroup$ Actually, the outputs are of different OSs and devices which vary a lot, and new OS/device are added on monthly bases. Would you still recommend regex based approach? We are currently using regex for these by which we can automate data generation, but would that be helpful? $\endgroup$ Dec 22, 2018 at 1:53
  • $\begingroup$ Based on what you say, predictions would be based in what regex can already do, in order to gather new data from new operating systems then you need to update the regex to get more data which in turn would already solve what you are trying to predict next. Even if the volume of logs is massive enough to justify a ML model performance will suffer compared to extraction with Regex. Check the edit on the answer for more information. $\endgroup$
    – wacax
    Dec 22, 2018 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.