Evaluate Keras model with scikit-learn metrics

Question

How does Keras calculate accuracy, precision, recall, and AUC? I've created a model for categorical classification (i.e., multiple classes) by using keras.losses.CategoricalCrossentropy() as loss function, and in the model.compile() method I've specified the following metrics:

METRICS = [
    keras.metrics.CategoricalAccuracy(name='acc'),
    keras.metrics.Precision(name='precision'),
    keras.metrics.Recall(name='recall'),
    keras.metrics.AUC(name='auc'),
]

When I evaluate the model with model.evaluate(X_test, y_test, verbose=2) the result is the following:

147/147 - 1s - loss: 0.5127 - acc: 0.8073 - precision: 0.8437 - recall: 0.7696 - auc: 0.9537

Then, I want to compare its performance with the results obtained with other algorithms (e.g., DecisionTreeClassifier() implemented in scikit-learn).

To evaluate the other classifiers, I'm using the following:

from sklearn.metrics import balanced_accuracy_score
from sklearn.metrics import roc_auc_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score

print("AUC: %2.3f" % roc_auc_score(y_test, pred_bin))
print("ACC: %2.3f" % balanced_accuracy_score(lb.inverse_transform(y_test), pred))
print("PRE: %2.3f" % precision_score(lb.inverse_transform(y_test), pred, average='weighted'))
print("REC: %2.3f" % recall_score(lb.inverse_transform(y_test), pred, average='weighted'))

However, when I test the performance of my NN with this piece of code, I obtain the following results:

ACC: 0.715 PRE: 0.801 REC: 0.807 AUC: 0.920

As you can note, the result differ from those obtained with model.evaluate() of Keras. How can I fairly compare the different methods?

Answer 1

You could use class KerasClassifier from keras.wrappers.scikit_learn, which wraps a Keras model in a scikit-learn interface, so that it can be used like other scikit-learn models and then you could evaluate it with scikit-learn's scoring functions, e.g.:

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.metrics import roc_curve, auc

keras_model = ...
classifier = KerasClassifier(keras_model, batch_size=32)
y_pred_keras = classifier.predict(X_test).ravel()
fpr, tpr, thresholds = roc_curve(y_test, y_pred_keras)
auc_score = auc(fpr, tpr)

You can find more info on the Keras docs, or googling around (e.g. this and this).