Problem building a feature vector

Question

I am trying build a classifier for malware analysis for which basing in the instructions of an assembly code, such as push, mov,... I want to predict the compiler, and in a second time the optimization op, and I am having some troubles. My code is the following:

#pakages
import numpy as np
import pandas as pd
import json as j
import re
import nltk
from nltk.tokenize import word_tokenize

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import *
from sklearn.metrics import confusion_matrix, classification_report
from sklearn import svm

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline


json_data = None;
with open('training_dataset.jsonl') as data_file:
    lines = data_file.readlines()
    joined_lines = "[" + ",".join(lines)+"]"

    json_data = j.loads(joined_lines)   


data = pd.DataFrame(json_data)
data.head()

which gives:

now, when I look at:

len(data['instructions'])

I have as output : 30000

but if I do the following:

for value in data['instructions'].iteritems():
    myList = list(value[1]);

myList

opcodes = [instruction.split()[0] for instruction in myList]

len(opcodes)

I get as output : 151

Why don't I have an output 30000? I don't understand why I have less elements. I want to use the opcodes to build a feature vector, but don't understand why the number of elements become so low.

Can somebody help me? Thank's in advance.

[EDIT] if it can be useful, if I do:

data['instructions']

I get as output:

Answer 1

Analyse this code

for value in data['instructions'].iteritems(): # Here we iter by line int the instruction colunm
    myList = list(value[1]); # Here you get 1 of the values and assing to your list

myList

opcodes = [instruction.split()[0] for instruction in myList]

len(opcodes)

your "myList" variable only holds 1 of the lines (the last one) of your data, which is probably the number of instructions in that line of your dataframe

While the 30k (I don't know how panda works here) is probably the number of lines in that column (or very unlikely the sum of all instructions in every line)