How does Keras calculate accuracy, precision, recall, and AUC?
I've created a model for categorical classification (i.e., multiple classes) by using keras.losses.CategoricalCrossentropy()
as loss function, and in the model.compile()
method I've specified the following metrics:
METRICS = [
keras.metrics.CategoricalAccuracy(name='acc'),
keras.metrics.Precision(name='precision'),
keras.metrics.Recall(name='recall'),
keras.metrics.AUC(name='auc'),
]
When I evaluate the model with model.evaluate(X_test, y_test, verbose=2)
the result is the following:
147/147 - 1s - loss: 0.5127 - acc: 0.8073 - precision: 0.8437 - recall: 0.7696 - auc: 0.9537
Then, I want to compare its performance with the results obtained with other algorithms (e.g., DecisionTreeClassifier()
implemented in scikit-learn).
To evaluate the other classifiers, I'm using the following:
from sklearn.metrics import balanced_accuracy_score
from sklearn.metrics import roc_auc_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score
print("AUC: %2.3f" % roc_auc_score(y_test, pred_bin))
print("ACC: %2.3f" % balanced_accuracy_score(lb.inverse_transform(y_test), pred))
print("PRE: %2.3f" % precision_score(lb.inverse_transform(y_test), pred, average='weighted'))
print("REC: %2.3f" % recall_score(lb.inverse_transform(y_test), pred, average='weighted'))
However, when I test the performance of my NN with this piece of code, I obtain the following results:
ACC: 0.715 PRE: 0.801 REC: 0.807 AUC: 0.920
As you can note, the result differ from those obtained with model.evaluate()
of Keras.
How can I fairly compare the different methods?