Regression using gradiant boosting - smoother predictions

Question

I have practical machine learning problem. I have trained a LightGBM model to predict house prices. Compared with other models I have tried, the loss (RMSE) is quite low and overall I'm quite happy with the predictions.

When investigating the predictions by tweaking features, I have noticed one problem. The trend is correct thus that when I increase size of house, the price generally goes up. However, in many cases a small change in a feature can result in prediction change in the wrong direction. Let's say all features remain the same but I increase house size a little and the predicted price goes down for example. If I increase it further it goes up again.

Are there any some techniques to handle this problem, and get "smoother", more continuous predictions?

Answer 1

Another approach could be to train several models and then simply take the average of their predictions - essentially using an ensemble. It will be necessary to initialise the models differently and ensure there is some randomness/stochasticity during training such that the models are indeed different.

This approach should allow your grupo of models to tolerate one or two models giving odd predictions in each new prediction, while still giving the most sensible result.

Studies show that this can dramatically smooth predictions. Have a look at this comparison of use cases and various ensemble technique from Armstrong. There are some guiding principles suggested, depending on saveral factors such as domain knowledge and variance of predictions.

Answer 2

Your algorithm might suffer from variance problem due to very low bias. I recommend you to act with regularization. And By this I mean using L1 regularization and adjusting other tuning parameters, especially the tree depth. There are actually some approaches belonging to explainable machine learning like the one you used( changing a bit a feature). Try the LIME approach.