0
$\begingroup$

In a regression problem that I'm currently working on, it seems that my model is doing well on higher values but significantly worse on lower values (e.g. values from 100,000,000 to 105,000,000 are being accurately predicted/ having lower error scores while values from 1,000,000 to 5,000,000 don't).

One approach that I am planning to test out is using multiple regression models, with one trained on the lower values and one on the higher values. I've seen scikit-learn's VotingRegressor, but if I understand correctly it seems that in predicting the value it'll only average the result from the estimators.

Other than using average values from the estimators, are there any other approaches to do the voting from multiple regression models? Since classification problems might use soft/hard voting, wondering if there are alternative approaches in regression problems as well.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You may try a stacking or blending approach (such as a StackingRegressor() in the recent sklearn versions), featuring a simple meta-model taking your initial models' predictions as features.

Improve this answer
$\endgroup$
3
  • $\begingroup$ If my understanding is correct, what it does is getting the predicted values from all estimators and use those values as features to predict the final predicted value? Would it make sense to use stackingregressor on top of ensemble estimators like RandomForest, XGBoost, etc? $\endgroup$
    – strivn
    Jul 14, 2022 at 5:45
  • $\begingroup$ Yes, stacking is a way to combine strong estimators. A stacking ensemble should be built it on top of diverse models: it is generally recommended to use different algorithms (like random forest + xgboost), but different data subsamples or hyperparameters should work as well. You can even stack neural networks trained on augmented and non-augmented data, for example. The meta-model should be kept as simple as possible (e.g. LinearRegression()). From my personal experience, ExtraTreesRegressor() often works better than RandomForestRegressor() for stacking, but YMMV. $\endgroup$
    – dx2-66
    Jul 14, 2022 at 7:40
  • $\begingroup$ I see, thanks a lot for sharing! $\endgroup$
    – strivn
    Jul 14, 2022 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.