Ensemble modelling using model's probabilities

Question

In a classification project, on the training sets, I ran a selection of classifiers. These give me about 20-30% accuracy at best. For each sample, I generate probabilities of each class. I want to make a new model which takes these probabilities and gives a weighted average of the probabilities which is accurate. For this, I tried averaging probabilities. I also tried taking the best probabilities from each model (For instance, assuming Class A has better precision/recall for model1 output, I take model1's outputs for class A and so on). This also hasn't improved the accuracy much. Can you suggest some ensembling techniques?

Answer 1

Let's say you fit n models on given dataset, to put them together, you have in fact few options:

• Take the average (I'd play with weighted or geometric)
• Voting (soft or hard). Basically you predict a class instead of probability and take the one which is most often predicted (hard) or take argmax or probabilities (soft). You havent mentioned what language you are using, but scikit has nice documentation for that.
• Fit a metaclassifier on these predictions. You have n models, hence n predictions for each record, so you fit another model on these predictions. This is called stacking as mentioned in the comment.
• You can go further for blending, boosting or bagging, there is really nice article about these methods here.