I've got a problem, which I thought could be solved by using a neural network:

I've got a binary file and a tool that converts that file into a readable html file (probably a text file as well). How the tool is converting that file is a secret and it is too complex to reverse engineer (>100000 lines of assembler for this function) and the binary files are too big and complex to interpret for a human in a feasible time. My idea was to teach a neural network with those binary files and the known output to train the network, how to convert these files. The binary files differ extremely in size(from a few kb up to serval mb).

I've got some basic knowledge of neural networks but I'm not sure where to start to search for the right approach. A friend of mine (with a bit more basic knowledge) said this is definitely doable and suggested a seq2seq model in tensorflow, but after some research I'm not sure if this is the right one for my problem.

In short your friend is correct, seq2seq is a reasonable match to the problem.

However, from your numbers this could be too complex to use current machine learning libraries on.

Despite you calling it "not feasible", you are far better off with the reverse engineering effort in my opinion.

If you really want to have a try using ML, you could start with some shorter files with the generated output and attempt to train a LSTM-based seq2seq model to match them. It may help, if the HTML is strongly templated and driven by data in the binary file, to reverse-engineer just the templating, and use seq2seq model to derive a more raw data sequence as opposed to HTML.

You should expect to spend several weeks learning how seq2seq works, followed by an unknown amount of time in the attempt to apply it to your problem. You will need a large number of sample files to train your model - hundreds of thousands, maybe millions. There might be exceptions if the generated HTML is very repetitive/formulaic, or if short segments of binary data can also generate short segments of HTML (so you could train on smaller segments of the larger files).


If any of this would break the terms and conditions for use of the software, I advise you not to attempt either reverse engineering or machine learning.

  • Thanks for that great anwser. I already had the fear that the required quantities for the files to train would be very high. The output is splitable into squences but I don't know where in the binary file. It seems like I have to go back to the drawing board. – Gistiv Oct 11 at 9:15

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.