Well, you can use this approach for stacking (AFAR this is how the process you are trying to perform is called):
1) Prepare a dataset in the format:
- Your original features which you used to fit the 4 estimators (remove the target variable from it)
- Add new target variable in it - the categorical output which shows which model should be picked up for that specific input. To obtain it - train your 4 estimators on your input and than label each input with the number of the estimator which has the closest output to the target
2) Train your stacker - on that dataset from #1 - for example the decision tree. Or any other classifier which will be able to accurately predict the number of the estimator which should be used for that specific input.
3) When using in combat mode - first run the stacker trained in #2 on new input and obtain an id of an estimator which best suites this example. Than run that estimator against new input record and get your response.
Hope that makes sense.
I tried it and it worked nicely for one of my problems.
Also you could try to employ some kind a 'blending' of these 4 predictors. Here is an approach you could take (my usage of terminology here becomes a bit frivolous since I am an engineer and not a scientist, so if someone provides a more elaborate answer on blending - please do).
First of all, recall how ANNs for classification work - and would work for you example, if you will use ANN as a stacker. You have input layer which reads the input values, some hidden layers and an output layer with a set of neurons, each neuron is associated with a specific class. For our case it would be 4 neurons. After you feed input each of 4 output neurons will output some real value, for example:
Neuron 1 - 5.4
Neuron 2 - 0.3
Neuron 3 - 15.3
Neuron 4 - 3.3
That output means that ANN thinks that the input belongs to 'class 3' and that you should use 3rd predictor for that example to get best results.
But! Let's think about probabilities. What is the probability of input to belong to ANY of 4 classes? It is 1.0 (since you have only 4 classes). And by default ANN doesn't have the knowledge that this the case. So for us it would be good if the ANN will output probability of data belong to X class instead of just some real value. For example we would like to obtain such output:
Neuron 1 - 0.6
Neuron 2 - 0.2
Neuron 3 - 0.15
Neuron 4 - 0.05
Which will read as 'I am 60% sure that you should use estimator 1 for that input, 20% sure - that estimator 2' etc. Such objective is called 'cross-entropy'. Modern libraries for building ANNs support it and you could tell network to train to output these probabilities.
By now I believe you already got what I suggest to do. Prepare the dataset as I described at the beginning of the answer, than train an ANN with cross-entropy objective to output the probabilities. When you run it in combat mode:
1) For new input run it through ANN and obtain probabilities, like these:
Neuron 1 - 0.6
Neuron 2 - 0.2
Neuron 3 - 0.15
Neuron 4 - 0.05
2) Run each predictor against input and obtain their predictions
3) Multiply each prediction by the associated probability and sum the results:
predictor1_result*0.6 + predictor2_result*0.2 + predictor3_result*0.15 + predictor4_result*0.05
And get you final result.
Why do that? First of all, we can't be sure that each time we realy pick up the best estimator, since our stacker (ANN in our case) could make mistakes. By using such approach we give the stacker a chance to compensate for possible mistakes - which minimizes the average prediction error.
Also, consider that you get such output from ANN:
Neuron 1 - 0.45
Neuron 2 - 0.45
Neuron 3 - 0.05
Neuron 4 - 0.05
In such case the best option for us is to almost average the predictions from predictor 1 and 2 - since ANN is not really sure which of predictors to use.
