# Voting Regression models, other approaches than averaging the results from each estimators

In a regression problem that I'm currently working on, it seems that my model is doing well on higher values but significantly worse on lower values (e.g. values from 100,000,000 to 105,000,000 are being accurately predicted/ having lower error scores while values from 1,000,000 to 5,000,000 don't).

One approach that I am planning to test out is using multiple regression models, with one trained on the lower values and one on the higher values. I've seen scikit-learn's VotingRegressor, but if I understand correctly it seems that in predicting the value it'll only average the result from the estimators.

Other than using average values from the estimators, are there any other approaches to do the voting from multiple regression models? Since classification problems might use soft/hard voting, wondering if there are alternative approaches in regression problems as well.

You may try a stacking or blending approach (such as a StackingRegressor() in the recent sklearn versions), featuring a simple meta-model taking your initial models' predictions as features.
• Yes, stacking is a way to combine strong estimators. A stacking ensemble should be built it on top of diverse models: it is generally recommended to use different algorithms (like random forest + xgboost), but different data subsamples or hyperparameters should work as well. You can even stack neural networks trained on augmented and non-augmented data, for example. The meta-model should be kept as simple as possible (e.g. LinearRegression()). From my personal experience, ExtraTreesRegressor() often works better than RandomForestRegressor() for stacking, but YMMV. Jul 14, 2022 at 7:40