1
$\begingroup$

How does Keras calculate accuracy, precision, recall, and AUC? I've created a model for categorical classification (i.e., multiple classes) by using keras.losses.CategoricalCrossentropy() as loss function, and in the model.compile() method I've specified the following metrics:

METRICS = [
    keras.metrics.CategoricalAccuracy(name='acc'),
    keras.metrics.Precision(name='precision'),
    keras.metrics.Recall(name='recall'),
    keras.metrics.AUC(name='auc'),
]

When I evaluate the model with model.evaluate(X_test, y_test, verbose=2) the result is the following:

147/147 - 1s - loss: 0.5127 - acc: 0.8073 - precision: 0.8437 - recall: 0.7696 - auc: 0.9537

Then, I want to compare its performance with the results obtained with other algorithms (e.g., DecisionTreeClassifier() implemented in scikit-learn).

To evaluate the other classifiers, I'm using the following:

from sklearn.metrics import balanced_accuracy_score
from sklearn.metrics import roc_auc_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score

print("AUC: %2.3f" % roc_auc_score(y_test, pred_bin))
print("ACC: %2.3f" % balanced_accuracy_score(lb.inverse_transform(y_test), pred))
print("PRE: %2.3f" % precision_score(lb.inverse_transform(y_test), pred, average='weighted'))
print("REC: %2.3f" % recall_score(lb.inverse_transform(y_test), pred, average='weighted'))

However, when I test the performance of my NN with this piece of code, I obtain the following results:

ACC: 0.715 PRE: 0.801 REC: 0.807 AUC: 0.920

As you can note, the result differ from those obtained with model.evaluate() of Keras. How can I fairly compare the different methods?

$\endgroup$
1
$\begingroup$

You could use class KerasClassifier from keras.wrappers.scikit_learn, which wraps a Keras model in a scikit-learn interface, so that it can be used like other scikit-learn models and then you could evaluate it with scikit-learn's scoring functions, e.g.:

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.metrics import roc_curve, auc

keras_model = ...
classifier = KerasClassifier(keras_model, batch_size=32)
y_pred_keras = classifier.predict(X_test).ravel()
fpr, tpr, thresholds = roc_curve(y_test, y_pred_keras)
auc_score = auc(fpr, tpr)

You can find more info on the Keras docs, or googling around (e.g. this and this).

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the advice, I will definitely try it. Just a curiosity: why the results are not the same? $\endgroup$ – Mattia Campana May 19 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.