2
$\begingroup$

I have a binary classification problem(benign/malicious) and I have applied simple neural network with one hidden layer for solving the problem. I have 46 features in my dataset and for the hidden layer I am using 46/2. Also my data is not scaled so for it I am using standardscaler. I am getting accuracy of around 99.79% with the code. However the accuracy was unexpected, I was expecting around 93-94%, I am afraid that I am leaking some data or making some silly mistake.

def create_baseline():
    model = Sequential()
    model.add(Dense(23, input_dim=46, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasClassifier(build_fn=create_baseline, epochs=1000, batch_size=len(X), validation_split=0.15, verbose=0)))
pipeline = Pipeline(estimators)
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)
scores = cross_val_score(pipeline, X, y, cv=kfold)
print("%.2f%%" % (scores[1]*100))

I am using around 5871 malicious samples and 3488 benign samples.

Just to cross check I have implemented without cross validation, a very simple implementation, and I am getting ~99% accuracy in that also.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)

scale = StandardScaler(with_mean=0, with_std=1)
new_X_train = scale.fit_transform(X_train)
new_X_test = scale.transform(X_test)

model = Sequential()

model.add(Dense(23, input_dim=46, kernel_initializer="normal"))
model.add(PReLU(alpha_initializer='zero', weights=None))

model.add(Dense(1, kernel_initializer='normal'))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

history = model.fit(new_X_train, y_train, epochs=1000, batch_size=len(X_train), validation_split=0.15)

scores = model.evaluate(new_X_test, y_test)
print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

print(history.history.keys())

plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

The plots I am getting are: lossaccuracy

$\endgroup$
  • 1
    $\begingroup$ I think one good way to evaluate would also be to plot validation loss during training loss. $\endgroup$ – Nischal Hp Feb 17 '18 at 9:49
  • $\begingroup$ I guess I have done that only, have plot loss and accuracy on the training and validation set over training epochs, please check the second code. $\endgroup$ – mg9893 Feb 17 '18 at 10:05
  • $\begingroup$ @NischalHp my cross validation accuracy and normal accuracy is coming same. Also I have computed correlation between features and removed all features with correlation>=0.8, got 19 features, then with the output, I computed correlation and got 4 features with correlation>0.20 {-0.231764, -0.407881, 0.391524, 0.727126}, I think getting good correlation with output means the feature is good. Is it bad to have such features in dataset? $\endgroup$ – mg9893 Feb 17 '18 at 13:49
  • $\begingroup$ I'm voting to close this question as off-topic because cross-posted from CV $\endgroup$ – Dawny33 Feb 27 '18 at 6:30
1
$\begingroup$

Considering the nature of your problem (disease classification?), probably you have imbalanced data: more negative/benign w.r.t. positive/malicious. In such cases accuracy is not a good metric to assess the quality of a model.

Consider, as example, a dataset with 99 "negative" cases and only one "positive" case, and consider a classifier which predict "negative" all the times. The accuracy of such model is, actually, 99%:

accuracy formula

What you want to do in order to deal with unbalanced data is:

1) just use another metric, as ROC-AUC. This can be sufficient if you have a dataset not so much unbalanced, say up to 1:20. This metric is not officially supported in Keras, but you can find a custom callback which calculate the AUC score every end of epoch here.

2) if the imbalance is very big, say 1:1000, you need to balance the dataset. There are lots of techniques, the simplest one is to make copies of the minority class samples, as many as needed to reach the balance. Other smarter techniques exist, and some of then are included in imbalanced-learn for SKLearn.

If you want to have a better overview of techniques and metrics used in such cases, I suggest you this paper.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I dont think I have imbalanced data, as mentioned above, I am using around 5871 malicious samples and 3488 benign samples. $\endgroup$ – mg9893 Feb 18 '18 at 6:56
  • $\begingroup$ oh, right, I missed that row. Then probably your hypothesis is right, there is some kind of leakage in your data. The simplest way I know is the plot_importance method of XGBoost: it will show the most important features, and if the most important of them is by far on an higher position w.r.t. the other, then you find the source of leakage. A nice and new method that can be user with Keras is SHAP, you can find an example here $\endgroup$ – Vincenzo Lavorini Feb 18 '18 at 8:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.