I have access to a few thousand files binary files and I don't have the program that generated them. The files have been parsed and I also have access to the database where the parsed fields are stored. What techniques can be used to figure out the format of the files, given this information?
I have had to perform that type of operation on so-called
.DAT files. You need a good hexadecimal editor, that allows you to display
hex contents from binary, and some patience. And probably a digit-probe friend you can talk with, to help decipher the riddle.
Seek global headers and local ones, file ID, dates and try to locate numbers or ascii code from your database. With that, you can infer field storage length with a relative ease, as least as long they are not exotic, like multiple of 4 bits. Sometimes, there are dummy bits to fill blanks. If the file is somehow compacted (one chunk on 3 bits, the next on 17 bits), it can become quite complicated. Imagine you are a private investigator, challenge a geek friend. Performing such a task with another person can make it funnier.