We got several models with predictions. How can we compare scores of different models with each other?
We assume that we got xgboost models and scores distribution can be different for each model, so how can we compare scores?
I'm going to assume your using python and scikit-learn mostly because it has a method for providing model metrics.
from sklearn.metrics import classification_report # I presume that you've already trained a model and it's saved as xgb # X_test is your testing X data (NOT THE DATA YOU TRAINED ON!!!!) # Y_test is the corresponding correct values print('Accuracy score is: ',accuracy_score(Y_test, xgb.predict(X_test)) * 100) print(classification_report(Y_test, xgb.predict(X_test))) >>> Accuracy score is: 61.13989637305699 >>> precision recall f1-score support 0 0.38 0.98 0.55 47 1 0.99 0.49 0.66 146 micro avg 0.61 0.61 0.61 193 macro avg 0.68 0.74 0.60 193 weighted avg 0.84 0.61 0.63 193
As you can see there is lots of info available. The accuracy score is the % of correct predictions overall. And the classification report goes into more depth about how good the model is predicting each class more info here crossvalidated SE explanation.
You included that
probability-calibration tag, which is prescient: there are a few techniques, all called "probability calibration," which adjust the scores output by a model to better fit observed probabilities. After this, the scores should be close to representing real probabilities, and should therefore be directly comparable.
The most common methods are Platt scaling and isotonic regression. There is a third and more recent method, beta calibration, and there are a few more exotic ones around. The three ones I've named all fit to a new dataset a univariate function with inputs your model's scores and outputs the actual observed labels. Platt scaling fits a sigmoid function, beta calibration fits a parametric model that is more general than sigmoid, and isotonic fits a nonparametric, arbitrary non-decreasing function. XGBoost's outputs are biased away from 0 and 1, so the sigmoid is generally ill-suited, so in this case go with beta or isotonic (or find something else to your liking). Isotonic, being more well-known, has more open-source implementations.