I am trying build a classifier for malware analysis for which basing in the instructions of an assembly code, such as push, mov,... I want to predict the compiler, and in a second time the optimization op, and I am having some troubles. My code is the following:
#pakages import numpy as np import pandas as pd import json as j import re import nltk from nltk.tokenize import word_tokenize from sklearn.model_selection import train_test_split from sklearn.feature_extraction.text import CountVectorizer from sklearn.naive_bayes import * from sklearn.metrics import confusion_matrix, classification_report from sklearn import svm #for visualizing data import matplotlib.pyplot as plt import seaborn as sns; sns.set(font_scale=1.2) %matplotlib inline json_data = None; with open('training_dataset.jsonl') as data_file: lines = data_file.readlines() joined_lines = "[" + ",".join(lines)+"]" json_data = j.loads(joined_lines) data = pd.DataFrame(json_data) data.head()
now, when I look at:
I have as output : 30000
but if I do the following:
for value in data['instructions'].iteritems(): myList = list(value); myList opcodes = [instruction.split() for instruction in myList] len(opcodes)
I get as output : 151
Why don't I have an output 30000? I don't understand why I have less elements. I want to use the opcodes to build a feature vector, but don't understand why the number of elements become so low.
Can somebody help me? Thank's in advance.
[EDIT] if it can be useful, if I do:
I get as output: