I'm using the EMBER dataset (malicious and benign binaries). I used the built in EMBER code to vectorize the data and then filtered the unlabeled from the training data. https://github.com/endgameinc/ember
Now I'm building a very simply NN using TensorFlow and Keras and no matter what parameters I play with it seems that the accuracy approaches 50%. For a comparison the EMBER team get's 98% when using a Decision Tree (LGBM i think).
as an example this is one of the many things I've tried.
model = keras.Sequential() model.add(keras.layers.Dense(100, input_dim=ndim, activation='relu')) model.add(keras.layers.Dropout(0.5)) # model.add(keras.layers.Dense(100, activation='relu')) # model.add(keras.layers.Dropout(0.5)) model.add(keras.layers.Dense(1, activation='sigmoid')) model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy']) model.fit(X_train,y_train, epochs=10000, batch_size=100)
In this instance I reduced my dataset to 1000 items from 600K to attempt more epochs. But either way I seem to approach .50. Here's an example epoch result
Epoch 2588/10000 1000/1000 [==============================] - 0s 86us/step - loss: 7.6877 - acc: 0.5180
I feel like I must be missing something fundamental as at some point the training data should get better even if it's just due to over fitting. Also, when the model completes and I evaluate the test data my accuracy is also almost exactly .50
EDIT 1: How I got the data The files X_train.dat etc were generated via the code in the EMBER repository, and so should be correct as is.
data_dir = "ember"
#taken from current version of PEFeatureExtractor in Ember
ndim = 2351
#at the moment these are simply just copies of the original
X_train_path = os.path.join(data_dir, "X_train.dat")
y_train_path = os.path.join(data_dir, "y_train.dat")
X_train = np.memmap(X_train_path, dtype=np.float32, mode="readonly", shape=(900000, ndim))
y_train = np.memmap(y_train_path, dtype=np.float32, mode="readonly", shape=900000)
not_negative_one = lambda x: not x == -1
vector_func = np.vectorize(not_negative_one)
condition = vector_func(y_train)
print("The total number of entries should be 900k: and is {}\n after the filtering the total number should be 600k: and is {}".format(len(condition),len(new_condition)))
y_train = y_train[condition]
X_train = X_train[condition]
print("Both y_train and X_train should now have the lengths of 600k: X={} y={}".format(len(y_train),len(X_train)))
data_dir = "."
X_train_path = os.path.join(data_dir, "X_train_no_unlabeled.dat")
y_train_path = os.path.join(data_dir, "y_train_no_unlabeled.dat")
new_X_train = np.memmap(X_train_path, dtype=np.float32, mode="r+", shape=(600000, ndim))
new_y_train = np.memmap(y_train_path, dtype=np.float32, mode="r+", shape=600000)
new_X_train[:] = X_train[:600000]
new_y_train[:] = y_train[:600000]
new_y_train.flush()
new_X_train.flush()
The above code was run once, then the below is run every time
#taken from current version of PEFeatureExtractor in Ember
ndim = 2351
X_train_path = os.path.join(data_dir, "X_train_no_unlabeled.dat")
y_train_path = os.path.join(data_dir, "y_train_no_unlabeled.dat")
X_train = np.memmap(X_train_path, dtype=np.float32, mode="c", shape=(600000, ndim))
y_train = np.memmap(y_train_path, dtype=np.float32, mode="c", shape=600000)
#randomly permute data points
inds = np.random.permutation(X_train.shape[0])
X_train = X_train[inds]
y_train = y_train[inds]
inds = np.random.permutation(X_test.shape[0])
X_test = X_test[inds]
y_test = y_test[inds]
tmp_list_y_train = list(y_train)
print("The training data consists of {} items, with {} malicious and {} benign".format(len(tmp_list_y_train),tmp_list_y_train.count(1),tmp_list_y_train.count(0)))
EDIT 2: I just tried tho model with two digits of the MNIST data set, and the model works just fine. So the model itself seems to be fine, but there is something wrong with the data set. I'm uncertain what that could be at this time.