Learning Algorithm that decide which model gives better results for each testing instance

Question

Is their any existing Ensemble technique which uses subset of training data to predict which algorithm is better for predicting each instance of testing data?

Let's say we have N sized training set and K sized testing set in which a particular attribute needed to be predicted using the training set. But there are hundreds of algorithms and ways we can use. We can divide training set into two parts and train each model with first half and decide test on second half. Based on characteristics, we can decide which algorithm to use for real test cases (K sized set). As an example lets say dataset have an attribute named "temperature". Particular algorithm may work well when temperature is higher than 100 Celsius. We can then classify all the 100 degree or above instances to particular class. Then final prediction will be done based on that by with that model class trained with all N sized data.

What I am asking is that is their any existing method similar to that?

Answer 1

Applying any non-linear model in a model stacking approach should do what you want. In brief the approach is to take predictions from other models as new features, plus the original data and labels, then use them to train a meta-model. Read the link, it offers practical advice on how to do this within a k-fold validation framework, which will give it a much better chance of doing well.

Non-linear models that combine simpler units - e.g. neural networks and models that use multiple decision trees (e.g. xgboost) - already perform this kind of internal split during training. So if you are already using those, you might not gain such a big improvement over simpler ensemble techniques, such as taking a mean or weighted mean over models.

Answer 2

I do not know of a specific algorithm that does this for you however it would not be hard to build. If you have two prediction algorithms then you can use them both (on training data) to put it into two classes. Each class would represent which algorithm would perform better. You then use a third algorithm to do binary classification to see if you can predict which of the original algorithms you should use. If you can predict which algorithm to use then you are set. You will get much better results if your classification algorithm is similar to your two prediction algorithms. In that case they will handle all the features in a similar way.

Once trained you then classify your test set and run the algorithms suggested by the classification.