# Understanding why my binary classification is approaching 50% accuracy using TensorFlow and Keras

1. I'm using the EMBER dataset (malicious and benign binaries). I used the built in EMBER code to vectorize the data and then filtered the unlabeled from the training data. https://github.com/endgameinc/ember

Now I'm building a very simply NN using TensorFlow and Keras and no matter what parameters I play with it seems that the accuracy approaches 50%. For a comparison the EMBER team get's 98% when using a Decision Tree (LGBM i think).

as an example this is one of the many things I've tried.

model = keras.Sequential()
loss='binary_crossentropy',
metrics=['accuracy'])
model.fit(X_train,y_train, epochs=10000, batch_size=100)


In this instance I reduced my dataset to 1000 items from 600K to attempt more epochs. But either way I seem to approach .50. Here's an example epoch result

Epoch 2588/10000 1000/1000 [==============================] - 0s 86us/step - loss: 7.6877 - acc: 0.5180

I feel like I must be missing something fundamental as at some point the training data should get better even if it's just due to over fitting. Also, when the model completes and I evaluate the test data my accuracy is also almost exactly .50

EDIT 1: How I got the data The files X_train.dat etc were generated via the code in the EMBER repository, and so should be correct as is.

data_dir = "ember"
#taken from current version of PEFeatureExtractor in Ember
ndim = 2351
#at the moment these are simply just copies of the original
X_train_path = os.path.join(data_dir, "X_train.dat")
y_train_path = os.path.join(data_dir, "y_train.dat")
X_train = np.memmap(X_train_path, dtype=np.float32, mode="readonly", shape=(900000, ndim))
y_train = np.memmap(y_train_path, dtype=np.float32, mode="readonly", shape=900000)

not_negative_one = lambda x: not x == -1
vector_func = np.vectorize(not_negative_one)
condition = vector_func(y_train)
print("The total number of entries should be 900k: and is {}\n after the filtering the total number should be 600k: and is {}".format(len(condition),len(new_condition)))
y_train = y_train[condition]
X_train = X_train[condition]
print("Both y_train and X_train should now have the lengths of 600k: X={} y={}".format(len(y_train),len(X_train)))

data_dir = "."
X_train_path = os.path.join(data_dir, "X_train_no_unlabeled.dat")
y_train_path = os.path.join(data_dir, "y_train_no_unlabeled.dat")
new_X_train = np.memmap(X_train_path, dtype=np.float32, mode="r+", shape=(600000, ndim))
new_y_train = np.memmap(y_train_path, dtype=np.float32, mode="r+", shape=600000)
new_X_train[:] = X_train[:600000]
new_y_train[:] = y_train[:600000]

new_y_train.flush()
new_X_train.flush()


The above code was run once, then the below is run every time

#taken from current version of PEFeatureExtractor in Ember
ndim = 2351
X_train_path = os.path.join(data_dir, "X_train_no_unlabeled.dat")
y_train_path = os.path.join(data_dir, "y_train_no_unlabeled.dat")
X_train = np.memmap(X_train_path, dtype=np.float32, mode="c", shape=(600000, ndim))
y_train = np.memmap(y_train_path, dtype=np.float32, mode="c", shape=600000)

#randomly permute data points
inds = np.random.permutation(X_train.shape[0])
X_train = X_train[inds]
y_train = y_train[inds]
inds = np.random.permutation(X_test.shape[0])
X_test = X_test[inds]
y_test = y_test[inds]

tmp_list_y_train = list(y_train)
print("The training data consists of {} items, with {} malicious and {} benign".format(len(tmp_list_y_train),tmp_list_y_train.count(1),tmp_list_y_train.count(0)))


EDIT 2: I just tried tho model with two digits of the MNIST data set, and the model works just fine. So the model itself seems to be fine, but there is something wrong with the data set. I'm uncertain what that could be at this time.

• It would be useful to provide code with which you prepare your data as it is likely to be a cause of poor quality. – Mikhail Berlinkov Nov 8 '18 at 19:26
• Ok I can do that when I get home. As a description I’m letting EMBER vectorize it, then loading it up and filtering labels that are -1 (unlabeled data). That leaves me with 1 for malicious and 0 for benign. After my filtering I check and I have the advertised amount of labeled data. – bravosierra99 Nov 8 '18 at 19:29