Say I have a regression problem where I'd like to predict values ranging from 0 - 100.000 based on some predictors. A single XGBoost model achieves decent overall mean performance (measured by MAE) using 5-fold CV.
Then I looked at specific sub-ranges of the [0,100.000] response interval to see where the model performed better or worse. It turned out that especially in the beginning and the end, i.e. [0, 5.000] and [80.000, 100.000], the model performed worst.
So I took a second step and partitioned the response interval into several non-overlapping sub-intervals and trained a separate regression model for each, resulting in significant better performance when compared to the predictions coming from the first model.
Now, what I have is a set of trained regression models for several sub-intervals of the response with good predictive performance for the entire range of the response. I am wondering how to apply these on unseen cases:
Obviously, for these I do not know the actual value (as this is what I want to predict), hence I do not know which of the sub-models to apply to get the prediction.
I was thinking of training another model - this time a classifier - that first predicts the interval which the case is belonging to and then secondly applying the regression model that was specifically trained on that particular interval to get the final prediction of the response.
This approach would be some kind of a "stacked" model, where the first one (the classifier) makes a rough decision on the range of the response and based on this prediction "routes" the cases to the particular regression model that predicts the exact value.
I have to questions:
- Is this a common approach? Are there better ones? (I was searching a lot in the internet but did not find anything on that matter.)
- I imagine training/testing this "hierarchical" model would be complicated and time-consuming to implement, especially using CV. I could imagine that it could be done using the "pipeline" framework from scikit-learn. Are there already some reference implementations I could use?
Thanks in advance for your comments/suggestions.