Let us say, I have 4 classification models on a training data set of various examples. Now, I want to choose which 1 out 4 models (or what combination of the 4 models) should I choose to get the final probability for a particular test example. So, basically I want to create a meta model to get ultimate probabilities for all the test examples. I know I can combine all 4 models using regression but I am particularly interested in using a decision tree to combine all 4 models.

So, the meta decision tree model will have 4 features/variables (4 model probabilities) as per my current thoughts. Is this a right approach? Should I include more features in it?

Please explain how do I combine 4 models using a meta decision tree model to choose the best model out of 4 for a particular test example.


1 Answer 1


Well, you can use this approach for stacking (AFAR this is how the process you are trying to perform is called):

1) Prepare a dataset in the format:

  • Your original features which you used to fit the 4 estimators (remove the target variable from it)
  • Add new target variable in it - the categorical output which shows which model should be picked up for that specific input. To obtain it - train your 4 estimators on your input and than label each input with the number of the estimator which has the closest output to the target

2) Train your stacker - on that dataset from #1 - for example the decision tree. Or any other classifier which will be able to accurately predict the number of the estimator which should be used for that specific input.

3) When using in combat mode - first run the stacker trained in #2 on new input and obtain an id of an estimator which best suites this example. Than run that estimator against new input record and get your response.

Hope that makes sense. I tried it and it worked nicely for one of my problems.

Also you could try to employ some kind a 'blending' of these 4 predictors. Here is an approach you could take (my usage of terminology here becomes a bit frivolous since I am an engineer and not a scientist, so if someone provides a more elaborate answer on blending - please do).

First of all, recall how ANNs for classification work - and would work for you example, if you will use ANN as a stacker. You have input layer which reads the input values, some hidden layers and an output layer with a set of neurons, each neuron is associated with a specific class. For our case it would be 4 neurons. After you feed input each of 4 output neurons will output some real value, for example: Neuron 1 - 5.4 Neuron 2 - 0.3 Neuron 3 - 15.3 Neuron 4 - 3.3

That output means that ANN thinks that the input belongs to 'class 3' and that you should use 3rd predictor for that example to get best results.

But! Let's think about probabilities. What is the probability of input to belong to ANY of 4 classes? It is 1.0 (since you have only 4 classes). And by default ANN doesn't have the knowledge that this the case. So for us it would be good if the ANN will output probability of data belong to X class instead of just some real value. For example we would like to obtain such output:

Neuron 1 - 0.6 Neuron 2 - 0.2 Neuron 3 - 0.15 Neuron 4 - 0.05

Which will read as 'I am 60% sure that you should use estimator 1 for that input, 20% sure - that estimator 2' etc. Such objective is called 'cross-entropy'. Modern libraries for building ANNs support it and you could tell network to train to output these probabilities.

By now I believe you already got what I suggest to do. Prepare the dataset as I described at the beginning of the answer, than train an ANN with cross-entropy objective to output the probabilities. When you run it in combat mode:

1) For new input run it through ANN and obtain probabilities, like these: Neuron 1 - 0.6 Neuron 2 - 0.2 Neuron 3 - 0.15 Neuron 4 - 0.05

2) Run each predictor against input and obtain their predictions 3) Multiply each prediction by the associated probability and sum the results:

predictor1_result*0.6 + predictor2_result*0.2 + predictor3_result*0.15 + predictor4_result*0.05

And get you final result.

Why do that? First of all, we can't be sure that each time we realy pick up the best estimator, since our stacker (ANN in our case) could make mistakes. By using such approach we give the stacker a chance to compensate for possible mistakes - which minimizes the average prediction error.

Also, consider that you get such output from ANN: Neuron 1 - 0.45 Neuron 2 - 0.45 Neuron 3 - 0.05 Neuron 4 - 0.05

In such case the best option for us is to almost average the predictions from predictor 1 and 2 - since ANN is not really sure which of predictors to use.

  • $\begingroup$ Hi, thanks for giving a detailed answer. Your first approach (creating new target variable to show which model is the closest to the original label) sounds really interesting. How did you come up with this idea? Did you read about it somewhere? If yes, please provide me with links to the sources, that would be really be helpful. Also, what does AFAR stands for? $\endgroup$ Oct 22, 2015 at 6:33
  • 1
    $\begingroup$ @user2409011 AFAR means 'as far as I recall' :) That approach is called "stacking" and is a typo of an ensemble learning. I suggest a book: web.stanford.edu/~hastie/local.ftp/Springer/OLD/… $\endgroup$ Oct 22, 2015 at 6:36
  • $\begingroup$ :D I thought AFAR stands for some technique. Anyways, ensemble learning normally means combining models not picking 1 out of a number of models. For ensemble generally people do logistic regression. So that is why asked if you read somewhere about your approach (i.e. picking 1 out of a number of models by creating a new target variable). Does the stanford source specifically deals with your approach? $\endgroup$ Oct 22, 2015 at 6:41
  • $\begingroup$ @user2409011 yes, there is a whole chapter on that just search by keyword 'stacking'. Also keep in mind that ensemble doesn't always mean 'combinig model predictions', there are lots of versions of ensemble learning - bagging, boosting, stacking, blending, bayesian averaging etc. So it is a whole family of different methods, which combine models in SOME way. $\endgroup$ Oct 22, 2015 at 6:49
  • 1
    $\begingroup$ @user2409011 almost forgot - here is also a good article which has less math - more code and is more accurate in terminology that I was: mlwave.com/kaggle-ensembling-guide $\endgroup$ Oct 22, 2015 at 6:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.