8
$\begingroup$

I presently receive files from a device in a semi-csv format. I have a written a simple recursive descent parser for getting information out of these files. Every time the device updates firmware, I have a new version of the parser for the changes the update brings.

Down the road, we will be taking data from other devices, which means another parser and more updates to firmware. I'm wondering if I could define a basic structure of "this is the data I need" and use a neural network to get the parsed data without having to write a parser for each new file type that comes in.

Is this a pipe dream or is it a valid application of machine learning? I'm much more of a software engineer than I am a data scientist, but I'm starting to dip my toes into the machine learning realm.

Thanks in advance.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I can not really answer to the above (reputation) but we will need some additional information to help you. How do the csv files look (before and after changes). Without knowing there is a few techniques you could try out. First one that comes to mind are regular expressions. If you want to solve this with machine learning, than this might or might not be possible most likekly using NLP with somewhat deep networks. I can direct you to some videos on the topic but first off some examples would help. $\endgroup$
    – pAt84
    Feb 6, 2016 at 20:52
  • 5
    $\begingroup$ The short answer to your question is "no". I found it too hard to write a good answer though, because there are some things that might chip away at the problem or related parts (e.g. a neural network to classify a format so you know which parsers to try first is feasible). They would probably be more work than they are worth, unless your goal was to learn some ML. My recommendation to start would be to just use normal development techniques to add abstraction and code re-use to your parsers. $\endgroup$
    – Neil Slater
    Feb 6, 2016 at 22:18
  • $\begingroup$ It's good to get some confirmation of that. I had been coming to the same conclusion. Thanks! $\endgroup$
    – Myles
    Feb 7, 2016 at 1:50

3 Answers 3

Reset to default
6
$\begingroup$

I would answer the question at two levels. The first level is "can it be done using machine learning?" I would say that machine learning is essentially about learning. So given that you prepare sufficient examples of sample documents and the output to expect from those documents, you can train a network to learn the structure of documents and extract the relevant information. The more general form of extracting information from documents is a well-researched problem and is more commonly known as Information Retrieval. And it is not limited to just machine learning techniques, you can use Natural Language Processing tools as well. So, in its general form, it is actually being done in practice. Coming to the second level, "should you be doing it using machine learning?". I would agree to what @NeilSlater said. The better and more feasible approach would be to use good programming practices so that you can reuse parts of your parser as your dataset evolves.

Improve this answer
$\endgroup$
2
$\begingroup$

Most programming languages and markup languages have a relatively simple syntax, so it is not usually necessary to use machine learning techniques to parse these languages. In machine learning, the process of automatically learning a formal grammar is also known as grammar induction. Several adaptive parsers have been designed for this purpose.

Improve this answer
$\endgroup$
0
$\begingroup$

I had similar problem for my business: parsing tons of mails with different formatting and some of them slightly changing over the time. I did not solve it, but I use the service from http://www.scriptminer.com/ which is apparently using machine learning tools to parse these sort of semi-structured messages.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.