Building an efficient feature vector

Question

I am building a classifier for malware analysis, which predicts if I have a malware by looking at the intructions of an assembly code, such as push, mov,... and predicting the optimization method. Note that I am considering a json file. My code is the following:

#pakages
import numpy as np
import pandas as pd
import json as j
import re
import nltk
from nltk.tokenize import word_tokenize


from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import *
from sklearn.metrics import confusion_matrix, classification_report
from sklearn import svm

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

json_data = None;
with open('training_dataset.jsonl') as data_file:
    lines = data_file.readlines()
    joined_lines = "[" + ",".join(lines)+"]"

    json_data = j.loads(joined_lines)    

data = pd.DataFrame(json_data)
data.head()

myList = [];
for value in data['instructions'].iteritems():
    myList.extend(list(value[1]))

opcodes = [instruction.split()[0] for instruction in myList]

vect = CountVectorizer()
x = vect.fit_transform(opcodes)
a =vect.vocabulary_

X = list(a.values())
X_all = np.array(X).reshape(-1,1)



Y = list(data['opt'])
MlistY = Y[ :395]
y_all = np.array(MlistY)

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all, 
      test_size=0.2, random_state=15)

from sklearn.svm import SVC
model = SVC()

model.fit(X_train,y_train)

y_pred = model.predict(X_test)
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

model.score(X_test,y_test)  

so, what I did is doing a feature extraction where I counted the numner of times each instruction such as push,mov,... appearsin the training set, and use these numbers as feature vectors. After that I had to cut the column data['opt'] in such a way to have the same number of elements of X_all. Then I split the dataset and I used as model support vector machines.

My problem is that the accuracy is very low, infact it is : 0.4810126582278481

I think this method I just used is called bag of words, but it is not very efficient for my case.

I think this is due to the fact that the method I used to extract the features is very inefficient.

My idea is to try to do a vectorization such that I assign to each operator a number, for example:

push -->0 mov -->1 jmp -->2 edx -->3

and so on and build a feature vector like this. But I also would like to keep track of the order on which the operators occurs inside the feature vector.

Is there a way to do this?

I have not found a specific vectorizer that does this, so is there a way for doing this type of vectorization?

Thank's in advance.

[EDIT] To create such feature vector where I keep the order I tried the following:

opcodes_ordered = pd.factorize(opcodes)

opcodes_ordered_true = opcodes_ordered[0]

opcodes_ordered_true

which returns : rray([ 0, 0, 0, ..., 22, 3, 5], dtype=int64)

Now I create the feature vector and define a model:

X_all_2 = opcodes_ordered_true.reshape(-1,1)[:30000] #had to cut the vector 
                                                     #because y has 30000 
                                                     # elements

y_all_2 = list(data['opt'])

X_train_2, X_test_2, y_train_2, y_test_2 = train_test_split(X_all_2, 
y_all_2, test_size=0.2, random_state=15)

model_2 = SVC(kernel = 'sigmoid',gamma = 1.0)

model_2.fit(X_train_2,y_train_2)

y_pred_2 = model_2.predict(X_test_2)
print(confusion_matrix(y_test_2, y_pred_2))
print(classification_report(y_test_2, y_pred_2))

model_2.score(X_test_2,y_test_2) 

but accuracy is still very low, in fact I have an accuracy of :

0.4841666666666667

I don't know what to do now.

[EDIT] I also tried to reduce the number of features, but by doing so I only got a small improvement.

[EDIT 2] What also I have tried to do is the following:

opcodes_ordered = pd.factorize(opcodes)

opcodes_ordered_true = opcodes_ordered[0]

opcodes_ordered_true

which gives as output : array([ 0, 0, 0, ..., 22, 3, 5], dtype=int64)

X_all_2 = opcodes_ordered_true.reshape(-1,1)[:1000]

y_all_2 = list(data['opt'])[:1000]

X_train_2, X_test_2, y_train_2, y_test_2 = train_test_split(X_all_2, 
y_all_2, test_size=0.2, random_state=15)

model_2 = SVC(kernel = 'linear',gamma = 1.0)

model_2.fit(X_train_2,y_train_2)

y_pred_2 = model_2.predict(X_test_2)
print(confusion_matrix(y_test_2, y_pred_2))
print(classification_report(y_test_2, y_pred_2))

model_2.score(X_test_2,y_test_2) 

but I get as accuracy : 0.56

which is still low. Does anybdy know how could I have better accuracy? Thank's in advance.

[EDIT 3] I don't kow if I am doing it correctly but to see if the dataset is balanced or not, I looked how many times in a dataset I have optimization high (H) and optimizaion low (L), which is also what I would like to predict for new samples.

Sorry if I am not really precise but I just started with machine learning.

What I did is the following:

Y = list(data['opt'])

MlistY = Y

MlistY.count('L')

which returns : 17924

MlistY.count('H')

which returns: 12076

Moreover I have also tried to use TfidfVectorizer and what I did is:

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(smooth_idf=False, sublinear_tf=False,norm=None, 
analyzer='word') 

x = vectorizer.fit_transform(opcodes)

a = vectorizer.vocabulary_

X = list(a.values())

X_all = np.array(X).reshape(-1,1)

Y = list(data['opt'])[:395]

MlistY = Y

y_all = np.array(MlistY)

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all, 
test_size=0.3, random_state=15)

from sklearn.svm import SVC
model = SVC(kernel = 'linear',C= 1)

model.fit(X_train,y_train)

y_pred = model.predict(X_test)
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

model.score(X_test,y_test) 

and in this case the accuracy is : 0.5294117647058824

Moreover, if I print the classification report I find this:

in particular this is for the case of TfidfVectorizer.

Answer 1

J.D., welcome to Data Science Exchange.

First, let's start with the following question: Based on what target distribution are you saying that your accuracy is low? Every problem in modelling, Regression, Binary Classification, Multiclass Classification has a baseline so we can say that our model has an acceptable performance. Sometimes, even humans can be our baseline. Also, what is your dataset size?

Second, do you have a balanced dataset? What I mean is, after you calculate your baseline, you might have a class, let's say no malware code that dominates your dataset. You should assign this by using method for imbalanced datasets such as Oversampling and Undersampling. You can read more about here.

Lastly, let's ignore everything I wrote and focus on the problem. Yes, you are correct, Bag of Words will ignore order in your data since it will just count the appearance of each word. I will list a few things you can try:

• You are using SVC class from sklearn. From docs I see the default kernel is rbf, have you tried using linear? Also, you can use LinearSVC.
• Try out RandomForest models, they perform really good even in text datasets.
• From CountVectorizer class, you could vary ngram_range parameters. Basically, it will create features based on grams, so let's say you use a 3-gram approach, then you will have for your first row: push_r12_push_rbp counting as one feature.
• Also, you could try Tf-Idf Vectorizer. TF-IDF Vectorizer is based on an algorithm where not only the count of words is taken into account but also the appearance in each document. Putting in simple terms, if a specific word appears too much in your dataset it will have a high inverse value and decrease its feature value, since this word will not be useful to differentiate your class. You can read more about it here: https://www.quora.com/How-does-TF-IDF-work
• Lastly, but not less important, you could try using a more advanced technique, WordEmbeddings, for example. It is an algorithm that will create real vectors from your text taking in consideration the enclosing words for each command. It is a little more complicate than that, again, you can learn more here. Note that for word embeddings to work properly, you should not have a small dataset. As a code example you can use this notebook of mine as a guidance.

I hope this help.