0
$\begingroup$

I am trying to build a classifier that predicts the compiler given some operations of assembly code. Here is the pandas dataframe:

enter image description here

What I do is using a TfidfVectorizer and select the features that have most predictive power by doing:

tfidf_vectorizer=TfidfVectorizer(max_features=500)

so using max_features=500 to select the 500 features with the highest idf. The problem is that the accuracy is still low, infact it is around 0.69. I would like to arrive at least at 0.9, but I dont't know what else do.

I am using support vector machines and this gives me the accuracy of 0.69. I also tried random fores and I was around 0.75.

My code is the following:

from sklearn.feature_extraction.text import TfidfVectorizer 

tfidf_vectorizer=TfidfVectorizer(max_features=1000)
df_x = df['opcodes']


X_all = tfidf_vectorizer_vectors=tfidf_vectorizer.fit_transform(df_x)
y_all = df['compiler']


X_train, X_test, y_train, y_test = train_test_split(X_all, y_all, 
      test_size=0.2, random_state=15)

from sklearn import svm

model = svm.SVC(kernel='linear', C=1).fit(X_train,y_train)

y_pred = model.predict(X_test)

acc = model.score(X_test, y_test)    
print("Accuracy %.3f" %acc)

moreover the dataset is balanced, infact I have 3 compilers with 1000 samples each.

I don't know what other startegy to try to increase accuracy and get at 0.9.

Can somebody please help me? Thank's in advance.

$\endgroup$
  • 1
    $\begingroup$ Do you have any indication that accuracy of 0.9 is obtainable in your dataset or are you just striving for an arbitrary goal? $\endgroup$ – Djib2011 Nov 2 '19 at 7:00
  • $\begingroup$ I know it is reachable because in the past it has been reached 0.9 of accuracy in this problem, but I have been working on it for a while and i can't impprove my accuracy. I think it is all bout feature selection, but I don't see other ways to choose features. $\endgroup$ – J.D. Nov 2 '19 at 7:50
0
$\begingroup$

I'm not familiar with scikit but I'm assuming that TfidfVectorizer represents bag of words features right? By this I mean that it treats all the instructions in an instance as a set, i.e. doesn't take into account their sequential order.

I'm also not familiar with compilers but I'm guessing that the order of the instructions could be a relevant indication? I.e. a compiler may generate particular sequences of instructions.

Based on these remarks I would try to represent instances with n-grams of instructions rather than individual instructions. Then you can still use some kind of bag-of-ngrams representation, possibly with TFIDF, but I would start with simple binary or frequency features. A simple feature selection step with something like information gain might be useful.

[edit] N-grams take order into account locally. In a bag of words model, words (or instructions in your case) are considered individually of each other: for instance the sequence push, push, mov is the same as push, mov, push. With bigrams this sequence would be represented as (push,push), (push,mov) whereas the second one is (push,mov), (mov,push). This means two things:

  • Higher level of detail about the instance, which can help the model capture the relevant indications
  • More features so higher risk of overfitting (the model taking some random details as indication, which lead to errors on the test set).
$\endgroup$
  • $\begingroup$ Thank's for your answer. Can I ask you what do you mean by take into account the sequential ordering and why it is important? I have already heard of the importance of keeping the order, but I have never understood how to do it. $\endgroup$ – J.D. Nov 2 '19 at 12:53
  • $\begingroup$ I also included n-grams as you suggested, in the following way : 'tfidf_vectorizer=TfidfVectorizer(ngram_range=(2,2),max_features =1000)' , but the accuracy is still the same. I also used cross- validation, but nothing changes. $\endgroup$ – J.D. Nov 2 '19 at 13:30
  • $\begingroup$ Added explanations about n-grams in the answer. Do you mean that the accuracy is exactly the same or not much different? exactly the same is very unlikely and points to an error somewhere. One problem I can see is the feature selection based on tf-idf only, that's not very good. I would rather try 1) remove any n-gram with a frequency less than 2 or 3; 2) if there are still too many features, use information gain to select the best N features (this will take into account whether a feature is informative or not wrt the class, as opposed to tfidf only). $\endgroup$ – Erwan Nov 2 '19 at 13:50
  • $\begingroup$ additionally you could try to vary the n-gram range and number of features. personally i like to try decision trees in order to analyze the resulting tree, see if it makes sense and whether it shows specific problems. oh and accuracy is not precise enough, you should look at the confusion matrix for a more detailed evaluation. $\endgroup$ – Erwan Nov 2 '19 at 13:53

Not the answer you're looking for? Browse other questions tagged or ask your own question.