How does Keras calculate accuracy, precision, recall, and AUC?
I've created a model for categorical classification (i.e., multiple classes) by using
keras.losses.CategoricalCrossentropy() as loss function, and in the
model.compile() method I've specified the following metrics:
METRICS = [ keras.metrics.CategoricalAccuracy(name='acc'), keras.metrics.Precision(name='precision'), keras.metrics.Recall(name='recall'), keras.metrics.AUC(name='auc'), ]
When I evaluate the model with
model.evaluate(X_test, y_test, verbose=2) the result is the following:
147/147 - 1s - loss: 0.5127 - acc: 0.8073 - precision: 0.8437 - recall: 0.7696 - auc: 0.9537
Then, I want to compare its performance with the results obtained with other algorithms (e.g.,
DecisionTreeClassifier() implemented in scikit-learn).
To evaluate the other classifiers, I'm using the following:
from sklearn.metrics import balanced_accuracy_score from sklearn.metrics import roc_auc_score from sklearn.metrics import precision_score from sklearn.metrics import recall_score print("AUC: %2.3f" % roc_auc_score(y_test, pred_bin)) print("ACC: %2.3f" % balanced_accuracy_score(lb.inverse_transform(y_test), pred)) print("PRE: %2.3f" % precision_score(lb.inverse_transform(y_test), pred, average='weighted')) print("REC: %2.3f" % recall_score(lb.inverse_transform(y_test), pred, average='weighted'))
However, when I test the performance of my NN with this piece of code, I obtain the following results:
ACC: 0.715 PRE: 0.801 REC: 0.807 AUC: 0.920
As you can note, the result differ from those obtained with
model.evaluate() of Keras.
How can I fairly compare the different methods?