0
$\begingroup$

In a classification project, on the training sets, I ran a selection of classifiers. These give me about 20-30% accuracy at best. For each sample, I generate probabilities of each class. I want to make a new model which takes these probabilities and gives a weighted average of the probabilities which is accurate. For this, I tried averaging probabilities. I also tried taking the best probabilities from each model (For instance, assuming Class A has better precision/recall for model1 output, I take model1's outputs for class A and so on). This also hasn't improved the accuracy much. Can you suggest some ensembling techniques?

$\endgroup$
1
  • 1
    $\begingroup$ You can try stacking your models. $\endgroup$
    – enterML
    Mar 24, 2017 at 10:52

1 Answer 1

2
$\begingroup$

Let's say you fit n models on given dataset, to put them together, you have in fact few options:

  • Take the average (I'd play with weighted or geometric)
  • Voting (soft or hard). Basically you predict a class instead of probability and take the one which is most often predicted (hard) or take argmax or probabilities (soft). You havent mentioned what language you are using, but scikit has nice documentation for that.
  • Fit a metaclassifier on these predictions. You have n models, hence n predictions for each record, so you fit another model on these predictions. This is called stacking as mentioned in the comment.
  • You can go further for blending, boosting or bagging, there is really nice article about these methods here.
$\endgroup$
3
  • $\begingroup$ Thanks. I did check VotingClassifier of sklearn, but my estimators are Keras models. Somehow it doesn't accept multilabel output in VotingClassifier of sklearn. So I couldn't use it. $\endgroup$ Mar 26, 2017 at 12:43
  • $\begingroup$ Stacking as mentioned, I tried using a simple feedforward layer on top of these predictions of n models to get the final predictions. But it performed really bad, hardly around 10% accuracy. So I gave up on that idea. $\endgroup$ Mar 26, 2017 at 12:44
  • 1
    $\begingroup$ Well it could be totally possible that the ensemble perform worse than single model. Maybe consider different models in the ensemble. They should be as different as possible. $\endgroup$
    – HonzaB
    Mar 26, 2017 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.