In a classification project, on the training sets, I ran a selection of classifiers. These give me about 20-30% accuracy at best. For each sample, I generate probabilities of each class. I want to make a new model which takes these probabilities and gives a weighted average of the probabilities which is accurate. For this, I tried averaging probabilities. I also tried taking the best probabilities from each model (For instance, assuming Class A has better precision/recall for model1 output, I take model1's outputs for class A and so on). This also hasn't improved the accuracy much. Can you suggest some ensembling techniques?
1 Answer
Let's say you fit n models on given dataset, to put them together, you have in fact few options:
- Take the average (I'd play with weighted or geometric)
- Voting (soft or hard). Basically you predict a class instead of probability and take the one which is most often predicted (hard) or take argmax or probabilities (soft). You havent mentioned what language you are using, but scikit has nice documentation for that.
- Fit a metaclassifier on these predictions. You have n models, hence n predictions for each record, so you fit another model on these predictions. This is called stacking as mentioned in the comment.
- You can go further for blending, boosting or bagging, there is really nice article about these methods here.
-
$\begingroup$ Thanks. I did check VotingClassifier of sklearn, but my estimators are Keras models. Somehow it doesn't accept multilabel output in VotingClassifier of sklearn. So I couldn't use it. $\endgroup$ Mar 26, 2017 at 12:43
-
$\begingroup$ Stacking as mentioned, I tried using a simple feedforward layer on top of these predictions of n models to get the final predictions. But it performed really bad, hardly around 10% accuracy. So I gave up on that idea. $\endgroup$ Mar 26, 2017 at 12:44
-
1$\begingroup$ Well it could be totally possible that the ensemble perform worse than single model. Maybe consider different models in the ensemble. They should be as different as possible. $\endgroup$– HonzaBMar 26, 2017 at 17:48