0
$\begingroup$

Let's say I have 5 models cross-validated via leave-one-out strategy. I have the predictions and scores of each model.

Now, it's time to calculate the average for the set of 5 models - am I supposed to:

  • add up the 5 losses and divide them by 5?
  • Or average their probabilities for each prediction and use the average probability to calculate new metrics like an ensemble/ forest?
Improve this question
$\endgroup$
2
  • $\begingroup$ are you actually interested in getting an average performance value for your five models? Or, as usually, you are interested in selecting the best model to be used afterwards? $\endgroup$ – German C M Dec 29 '20 at 14:00
  • $\begingroup$ @GermanCM I'm writing a library, so I want to provide whatever functionality is most generally useful. So if people are just picking the best model and moving on, it feels like the average doesn't much matter? $\endgroup$ – HashRocketSyntax Dec 29 '20 at 14:02
1
$\begingroup$

A standard way to provide the performance of each model would be:

  • providing, for each split, the value of the chosen metric (accuracy, roc_auc, etc) on the train and test sets (on your case, your one-out sample), something like this (in this case with 2 models): enter image description here

  • as a final model performance (for each one of the 5 models), a mean metric value together with its standard deviation for the test sets is a way to inform about the model quality and its robustness, something like (preferably for the test set):

enter image description here

You have more detail on how to automatically get this done via scikit-learn, and in this answer and this one.

By the way, consider using another strategy as stratified k-fold, in case you have a lot of samples, as leave-one-out would be very costly.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Okay for each model you want split stats, I get that. But wouldn't you want to know the average across all of those individual models? $\endgroup$ – HashRocketSyntax Dec 29 '20 at 14:44
  • $\begingroup$ Depending on your goal, do you want eventually to select the best model out of those 5 ones? Then, the answer is that you do not need the average across the 5 models; otherwise, do you want to build a final ensemble model? $\endgroup$ – German C M Dec 29 '20 at 15:16
0
$\begingroup$

There are multiple popular ways to ensemble models. Averaging, majority voting, selecting the one with the highest probability, learn a new model based on these 5 numbers are amongst the many methods available. Check also the Bayes optimal classifier which 'averages' these probabilities in a Bayes way: https://en.wikipedia.org/wiki/Ensemble_learning#Bayes_optimal_classifier

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.