1
$\begingroup$

Given a sample of hexadecimal data, I would like to identify UNKNOWN sequences of bytes that are repeated throughout the sample. (Not searching for a known string or value) I am attempting to reverse engineer a network protocol, and I am working on determining data structures within the packet. As an example of what I'm trying to do (albeit on a smaller scale):

(af:b6:ea:3d:83:02:00:00):{21:03:00:00}:[b3:49:96:23:01]

{21:03:00:00}:(af:b6:ea:3d:83:02:00:00):01:42:00:00:00:00:01:57

And

(38:64:88:6e:83:02:00:00):{26:03:00:00}:[b3:49:96:23:01]

{26:03:00:00}:(38:64:88:6e:83:02:00:00):01:42:00:00:00:00:00:01

Obviously, these are easy to spot by eye, but patterns that are hundreds of chars into the data are not. I'm not expecting a magic bullet for the solution, just a nudge in the right direction, or even better, a premade tool.

I'm currently needing this for a C# project, but I am open to any and all tools.

$\endgroup$
3
$\begingroup$

I believe the problem that you are referring to, is that of "Motif Discovery in Time Series Data". An appreciable amount of research literature already exists in this domain, so you can look through those. If the data that you handle is not very large, you can find some relatively easy to implement algorithms.

If the data is large , you can look at more recent publications in this domain. As a starting point I would recommend taking a look at how Motif Discovery is done in SAX. SAX takes continuous signals as inputs and discretizes them. These discrete levels are then stored as alphabets. This resulting data looks very much like yours in my opinion. Take a look at what they do in "Mining Motifs in Massive Time Series Databases".

$\endgroup$
1
  • $\begingroup$ Looks like what I'm trying to do. Thanks for the tip! $\endgroup$
    – Ron Brogan
    Jul 10 '15 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.