0
$\begingroup$

We got several models with predictions. How can we compare scores of different models with each other?

We assume that we got xgboost models and scores distribution can be different for each model, so how can we compare scores?

$\endgroup$
  • $\begingroup$ You have various models and you want to compare their scores? But you already have the scores, so the metric has already been chosen and the score calculated. Is that right? If so, you have one score per trained model. How many models do you have (that you need some method to compare scores, which are just numbers) and what exactly do you mean by comparing them? $\endgroup$ – 89f3a1c Aug 14 at 13:10
  • $\begingroup$ @89f3a1c, OP: or do you mean the scores given to each sample? $\endgroup$ – Ben Reiniger Aug 14 at 13:17
  • $\begingroup$ scores for each model can be distributed differently so you can't really say that 0.8 in one model means he is more likely to buy then if he has 0.78 in other model $\endgroup$ – Vladimir Ershov Aug 14 at 16:24
  • $\begingroup$ You've used the probability-calibration tag, so presumably you know a little about the standard methods for that? $\endgroup$ – Ben Reiniger Aug 14 at 18:22
  • $\begingroup$ This question could be improved by helping us understand what specific kind of comparison you're looking for. $\endgroup$ – Thomas Cleberg Aug 15 at 13:14
0
$\begingroup$

I'm going to assume your using python and scikit-learn mostly because it has a method for providing model metrics.

from sklearn.metrics import classification_report

# I presume that you've already trained a model and it's saved as xgb
# X_test is your testing X data (NOT THE DATA YOU TRAINED ON!!!!)
# Y_test is the corresponding correct values
print('Accuracy score is: ',accuracy_score(Y_test, xgb.predict(X_test)) * 100)
print(classification_report(Y_test, xgb.predict(X_test)))
>>> Accuracy score is:  61.13989637305699
>>>           precision    recall  f1-score   support

           0       0.38      0.98      0.55        47
           1       0.99      0.49      0.66       146

   micro avg       0.61      0.61      0.61       193
   macro avg       0.68      0.74      0.60       193
weighted avg       0.84      0.61      0.63       193

As you can see there is lots of info available. The accuracy score is the % of correct predictions overall. And the classification report goes into more depth about how good the model is predicting each class more info here crossvalidated SE explanation.

$\endgroup$
  • $\begingroup$ i have already trained 10 models(binary if he buy product1 or not. each model for each product not multiclass) now i have different distributions in scores that xgb returns. so i can't say for sure that if for product 1 person has score 0.8 he will more likely buy 1 product then product 2 where he has score 0.78 $\endgroup$ – Vladimir Ershov Aug 14 at 16:26
  • $\begingroup$ Why do you want to only have binary models? What are you actually trying to predict? $\endgroup$ – Tasty213 Aug 14 at 16:34
  • $\begingroup$ i want to predict if someone will buy a product. i have xgb scores as result. And i want to compare several models to see what product is more preferable $\endgroup$ – Vladimir Ershov Aug 15 at 13:14
  • $\begingroup$ Ok so if your only trying to predict which product someone will buy, why do you need multiple models? $\endgroup$ – Tasty213 Aug 15 at 13:29
  • $\begingroup$ because for each product i have different batches of people that can intersect and these models can be calculated in different time periods $\endgroup$ – Vladimir Ershov Aug 15 at 15:09
0
$\begingroup$

You included that probability-calibration tag, which is prescient: there are a few techniques, all called "probability calibration," which adjust the scores output by a model to better fit observed probabilities. After this, the scores should be close to representing real probabilities, and should therefore be directly comparable.

The most common methods are Platt scaling and isotonic regression. There is a third and more recent method, beta calibration, and there are a few more exotic ones around. The three ones I've named all fit to a new dataset a univariate function with inputs your model's scores and outputs the actual observed labels. Platt scaling fits a sigmoid function, beta calibration fits a parametric model that is more general than sigmoid, and isotonic fits a nonparametric, arbitrary non-decreasing function. XGBoost's outputs are biased away from 0 and 1, so the sigmoid is generally ill-suited, so in this case go with beta or isotonic (or find something else to your liking). Isotonic, being more well-known, has more open-source implementations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.