I am currently working on a problem where the dataset contains 200+ features (Let's call them the code features, e.g no.of.loops, memoryInst, loadInst, etc and Flags that are used to compile code that has such characteristics/code features)

The flags are represented as strings:

This is just dummy data.

snippet                 FlagsUsed                   no.of.loops   loadInst      memoryInst

1  Mergesort      " -a -b -c -d=10 -e -f =19  -c "    1              0             10

2  Bubblesort     " -a -c -f=230 "                    2              5             3

3  MatrixMulti     " -f=20 -z -f12 -f2f "             0              10            4 

I need some help with how these flags should be represented in the data, I have tried one-hot encoding and dummy_variable methods but these methods have some disadvantage:

1) One-hot encoding: using this method, the no. features will become huge, there are 150+ flags and I creating a one hot vector for each one of them would result in 22500 feature which is not feasible.

   flag1    flag2    flag3  flag4
1  -a       -b       -c      NA
2  -c       -a       -b      -z

I would need to create long vectors for each of the above feature flag1, flag2.. flag4

2) Dummy_variable method: There are 200+ different types/levels/factors of these flags and dummy_variable method create a feature for each level/factor which is not feasible.

Also, the flags can repeat in a single string (1st snippet, -c repeats).

I am thinking of some clever hashing that would maintain information regarding the sequence of the flags and there value (toggle flag = 0/1, threshold flag = {lower, upper} ). But the problem with hashing is, I have to, in future, predict these flags using other features (code features) and if I hash these flags somehow I won't be able to reverse hash them.

I am thinking of some fixed size vector representation which could be reversed so that I can tell flag using a numeric or hex number.

Can anyone please guide me or put me in the right direction. Would be thankful!

  • $\begingroup$ Why is it that your initial representation won't work? In other words, if you make all your flags one big string per observation, why would that not work? $\endgroup$ – grldsndrs Jul 17 '19 at 16:03
  • $\begingroup$ Each of the flag in the string has an effect on the performance of compiler, so I cannot use the flags string as it is. Also, in future, I would like to predict flags based on the code features and for that, the algo needs to learn the effect of flags and their relation to code features, this is not possible if I use big flag strings, right? Therefre I need to represent flags in such a way that the algo can see what flags were used in what positions with what code features. $\endgroup$ – Obiii Jul 18 '19 at 8:47
  • $\begingroup$ What are the possible combination to flag one $\endgroup$ – grldsndrs Jul 18 '19 at 10:58
  • $\begingroup$ each of the flag1, flag2, flag3 can have upto 150 different flags, and could repeat for example: flag1 flag2 flag3 1 -a -b -c 2 -b -d -e 3 -z NA NA 4 -a -a NA 5 -c -d -b The text in comments is not formated so It if you could plz copy paste and format the data in this comment in editor. $\endgroup$ – Obiii Jul 19 '19 at 8:43
  • $\begingroup$ Lets be simple and clear, tell me only about flag 1. How many combo's? $\endgroup$ – grldsndrs Jul 19 '19 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.