1. I'm using the EMBER dataset (malicious and benign binaries). I used the built in EMBER code to vectorize the data and then filtered the unlabeled from the training data. https://github.com/endgameinc/ember

    Now I'm building a very simply NN using TensorFlow and Keras and no matter what parameters I play with it seems that the accuracy approaches 50%. For a comparison the EMBER team get's 98% when using a Decision Tree (LGBM i think).

    as an example this is one of the many things I've tried.

    model = keras.Sequential()
    model.add(keras.layers.Dense(100, input_dim=ndim, activation='relu'))
    model.add(keras.layers.Dropout(0.5))
    # model.add(keras.layers.Dense(100, activation='relu'))
    # model.add(keras.layers.Dropout(0.5))
    model.add(keras.layers.Dense(1, activation='sigmoid'))
    model.compile(optimizer='adam',
                  loss='binary_crossentropy',
                  metrics=['accuracy'])
    model.fit(X_train,y_train, epochs=10000, batch_size=100)
    

    In this instance I reduced my dataset to 1000 items from 600K to attempt more epochs. But either way I seem to approach .50. Here's an example epoch result

    Epoch 2588/10000 1000/1000 [==============================] - 0s 86us/step - loss: 7.6877 - acc: 0.5180

    I feel like I must be missing something fundamental as at some point the training data should get better even if it's just due to over fitting. Also, when the model completes and I evaluate the test data my accuracy is also almost exactly .50

EDIT 1: How I got the data The files X_train.dat etc were generated via the code in the EMBER repository, and so should be correct as is.

data_dir = "ember"
#taken from current version of PEFeatureExtractor in Ember
ndim = 2351
#at the moment these are simply just copies of the original
X_train_path = os.path.join(data_dir, "X_train.dat")
y_train_path = os.path.join(data_dir, "y_train.dat")
X_train = np.memmap(X_train_path, dtype=np.float32, mode="readonly", shape=(900000, ndim))
y_train = np.memmap(y_train_path, dtype=np.float32, mode="readonly", shape=900000)

not_negative_one = lambda x: not x == -1
vector_func = np.vectorize(not_negative_one)
condition = vector_func(y_train)
print("The total number of entries should be 900k: and is {}\n after the filtering the total number should be 600k: and is {}".format(len(condition),len(new_condition)))
y_train = y_train[condition]
X_train = X_train[condition]
print("Both y_train and X_train should now have the lengths of 600k: X={} y={}".format(len(y_train),len(X_train)))

data_dir = "."
X_train_path = os.path.join(data_dir, "X_train_no_unlabeled.dat")
y_train_path = os.path.join(data_dir, "y_train_no_unlabeled.dat")
new_X_train = np.memmap(X_train_path, dtype=np.float32, mode="r+", shape=(600000, ndim))
new_y_train = np.memmap(y_train_path, dtype=np.float32, mode="r+", shape=600000)
new_X_train[:] = X_train[:600000]
new_y_train[:] = y_train[:600000]

new_y_train.flush()
new_X_train.flush()

The above code was run once, then the below is run every time

#taken from current version of PEFeatureExtractor in Ember
ndim = 2351
X_train_path = os.path.join(data_dir, "X_train_no_unlabeled.dat")
y_train_path = os.path.join(data_dir, "y_train_no_unlabeled.dat")
X_train = np.memmap(X_train_path, dtype=np.float32, mode="c", shape=(600000, ndim))
y_train = np.memmap(y_train_path, dtype=np.float32, mode="c", shape=600000)

#randomly permute data points
inds = np.random.permutation(X_train.shape[0])
X_train = X_train[inds]
y_train = y_train[inds]
inds = np.random.permutation(X_test.shape[0])
X_test = X_test[inds]
y_test = y_test[inds]

tmp_list_y_train = list(y_train)
print("The training data consists of {} items, with {} malicious and {} benign".format(len(tmp_list_y_train),tmp_list_y_train.count(1),tmp_list_y_train.count(0)))

EDIT 2: I just tried tho model with two digits of the MNIST data set, and the model works just fine. So the model itself seems to be fine, but there is something wrong with the data set. I'm uncertain what that could be at this time.

  • It would be useful to provide code with which you prepare your data as it is likely to be a cause of poor quality. – Mikhail Berlinkov Nov 8 at 19:26
  • Ok I can do that when I get home. As a description I’m letting EMBER vectorize it, then loading it up and filtering labels that are -1 (unlabeled data). That leaves me with 1 for malicious and 0 for benign. After my filtering I check and I have the advertised amount of labeled data. – bravosierra99 Nov 8 at 19:29
up vote 1 down vote accepted

You should consider analysing your data before applying NN. For instance, unlike Decision Trees Dense layers are not supposed to work well with denormalised features.

New contributor
Mikhail Berlinkov is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Try using a different activation function in your final layer (softmax activation should work better). If still you get same result, you could experiment by adding more layers, or increasing the number of neurons in your layers (maybe classifying your data is challenging, so working with such data needs more neurons), change optimizer or check with different learning rates and for now, till you get a decent training accuracy, remove dropout layer. Only when you are getting a decent training accuracy, consider adding dropout.

  • I did not find this answer helpful. You are recommending making small changes, when that is not the problem. Something is wrong with the model, as it's accuracy is 50% on a binary classification problem, and never gets better. Though in this case, it appears as though something is wrong with my data, as I just tested this model with two digits of MNIST and it works just fine. – bravosierra99 Nov 12 at 17:12
  • 1
    Actually, changing to softmax is not a little change for final activation but as you predict probability of one class this would be just wrong. However, you should consider analysing your data before applying NN. For instance, unlike Decision Trees Dense layers are not supposed to work well with denormalised features. – Mikhail Berlinkov Nov 12 at 19:32
  • Also, there are functions that cannot be predicted with 3-layer NN. – Mikhail Berlinkov Nov 12 at 19:34
  • @MikhailBerlinkov I have yet to test, but I think you have hit on the issue. I need to normalize my data. And yes you are correct softmax would be wrong – bravosierra99 Nov 13 at 20:16
  • 1
    @mikhailBerlinkov If you would like to create an answer that says I need to normalize my data, I will gladly mark it as correct. – bravosierra99 Nov 15 at 16:37

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.