I am trying build a classifier for malware analysis for which basing in the instructions of an assembly code, such as push, mov,... I want to predict the compiler, and in a second time the optimization op, and I am having some troubles. My code is the following:

import numpy as np
import pandas as pd
import json as j
import re
import nltk
from nltk.tokenize import word_tokenize

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import *
from sklearn.metrics import confusion_matrix, classification_report
from sklearn import svm

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

json_data = None;
with open('training_dataset.jsonl') as data_file:
    lines = data_file.readlines()
    joined_lines = "[" + ",".join(lines)+"]"

    json_data = j.loads(joined_lines)   

data = pd.DataFrame(json_data)

which gives:

enter image description here

now, when I look at:


I have as output : 30000

but if I do the following:

for value in data['instructions'].iteritems():
    myList = list(value[1]);


opcodes = [instruction.split()[0] for instruction in myList]


I get as output : 151

Why don't I have an output 30000? I don't understand why I have less elements. I want to use the opcodes to build a feature vector, but don't understand why the number of elements become so low.

Can somebody help me? Thank's in advance.

[EDIT] if it can be useful, if I do:


I get as output:

enter image description here


1 Answer 1


Analyse this code

for value in data['instructions'].iteritems(): # Here we iter by line int the instruction colunm
    myList = list(value[1]); # Here you get 1 of the values and assing to your list


opcodes = [instruction.split()[0] for instruction in myList]


your "myList" variable only holds 1 of the lines (the last one) of your data, which is probably the number of instructions in that line of your dataframe

While the 30k (I don't know how panda works here) is probably the number of lines in that column (or very unlikely the sum of all instructions in every line)

  • $\begingroup$ thank's for your answer. Can I ask you how do you see that I am considering only the last line? And do you know how could I consider other lines? My idea could be to do this procedure for all the lines and add each list to the other so to have all the values, but I don't know how to do it. Thank's in advance. $\endgroup$
    – J.D.
    Oct 28, 2019 at 5:17
  • $\begingroup$ The myList variable is overwritten at each instruction. If you want it to contain the full aggregated list, initialize it first: myList = [], then aggregate the data at each iteration: myList.extend(list(value[1])). $\endgroup$ Oct 28, 2019 at 6:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.