Understanding why my binary classification is approaching 50% accuracy using TensorFlow and Keras

Question

1. I'm using the EMBER dataset (malicious and benign binaries). I used the built in EMBER code to vectorize the data and then filtered the unlabeled from the training data. https://github.com/endgameinc/ember

   Now I'm building a very simply NN using TensorFlow and Keras and no matter what parameters I play with it seems that the accuracy approaches 50%. For a comparison the EMBER team get's 98% when using a Decision Tree (LGBM i think).

   as an example this is one of the many things I've tried.

   model = keras.Sequential()
   model.add(keras.layers.Dense(100, input_dim=ndim, activation='relu'))
   model.add(keras.layers.Dropout(0.5))
   # model.add(keras.layers.Dense(100, activation='relu'))
   # model.add(keras.layers.Dropout(0.5))
   model.add(keras.layers.Dense(1, activation='sigmoid'))
   model.compile(optimizer='adam',
                 loss='binary_crossentropy',
                 metrics=['accuracy'])
   model.fit(X_train,y_train, epochs=10000, batch_size=100)
   

   In this instance I reduced my dataset to 1000 items from 600K to attempt more epochs. But either way I seem to approach .50. Here's an example epoch result

   Epoch 2588/10000 1000/1000 [==============================] - 0s 86us/step - loss: 7.6877 - acc: 0.5180

   I feel like I must be missing something fundamental as at some point the training data should get better even if it's just due to over fitting. Also, when the model completes and I evaluate the test data my accuracy is also almost exactly .50

EDIT 1: How I got the data The files X_train.dat etc were generated via the code in the EMBER repository, and so should be correct as is.

data_dir = "ember"
#taken from current version of PEFeatureExtractor in Ember
ndim = 2351
#at the moment these are simply just copies of the original
X_train_path = os.path.join(data_dir, "X_train.dat")
y_train_path = os.path.join(data_dir, "y_train.dat")
X_train = np.memmap(X_train_path, dtype=np.float32, mode="readonly", shape=(900000, ndim))
y_train = np.memmap(y_train_path, dtype=np.float32, mode="readonly", shape=900000)

not_negative_one = lambda x: not x == -1
vector_func = np.vectorize(not_negative_one)
condition = vector_func(y_train)
print("The total number of entries should be 900k: and is {}\n after the filtering the total number should be 600k: and is {}".format(len(condition),len(new_condition)))
y_train = y_train[condition]
X_train = X_train[condition]
print("Both y_train and X_train should now have the lengths of 600k: X={} y={}".format(len(y_train),len(X_train)))

data_dir = "."
X_train_path = os.path.join(data_dir, "X_train_no_unlabeled.dat")
y_train_path = os.path.join(data_dir, "y_train_no_unlabeled.dat")
new_X_train = np.memmap(X_train_path, dtype=np.float32, mode="r+", shape=(600000, ndim))
new_y_train = np.memmap(y_train_path, dtype=np.float32, mode="r+", shape=600000)
new_X_train[:] = X_train[:600000]
new_y_train[:] = y_train[:600000]

new_y_train.flush()
new_X_train.flush()

The above code was run once, then the below is run every time

#taken from current version of PEFeatureExtractor in Ember
ndim = 2351
X_train_path = os.path.join(data_dir, "X_train_no_unlabeled.dat")
y_train_path = os.path.join(data_dir, "y_train_no_unlabeled.dat")
X_train = np.memmap(X_train_path, dtype=np.float32, mode="c", shape=(600000, ndim))
y_train = np.memmap(y_train_path, dtype=np.float32, mode="c", shape=600000)

#randomly permute data points
inds = np.random.permutation(X_train.shape[0])
X_train = X_train[inds]
y_train = y_train[inds]
inds = np.random.permutation(X_test.shape[0])
X_test = X_test[inds]
y_test = y_test[inds]

tmp_list_y_train = list(y_train)
print("The training data consists of {} items, with {} malicious and {} benign".format(len(tmp_list_y_train),tmp_list_y_train.count(1),tmp_list_y_train.count(0)))

EDIT 2: I just tried tho model with two digits of the MNIST data set, and the model works just fine. So the model itself seems to be fine, but there is something wrong with the data set. I'm uncertain what that could be at this time.

Answer 1

You should consider analysing your data before applying NN. For instance, unlike Decision Trees Dense layers are not supposed to work well with denormalised features.

Answer 2

Try using a different activation function in your final layer (softmax activation should work better). If still you get same result, you could experiment by adding more layers, or increasing the number of neurons in your layers (maybe classifying your data is challenging, so working with such data needs more neurons), change optimizer or check with different learning rates and for now, till you get a decent training accuracy, remove dropout layer. Only when you are getting a decent training accuracy, consider adding dropout.