3
$\begingroup$

I am trying to tune a Regression gradient boosting model where my target variable is zero inflated (80% zero) and the rest of the values are distributed as positive and negative values (not necessary symmetrically). What are good practices when training a model like this?

Any specific issues which I should be aware of to generate a good model? Based on my research, Tweedie Gradient Boosting is not fitted for this model because my target variable has a mix of negative and positive values around the zero mode; therefore, it doesn’t follow a tweedie distribution.

$\endgroup$
5
  • $\begingroup$ My first thought is that gradient boosted trees are sufficiently flexible that you can just treat this as regression without much customization. My second thought is that you might want a customized loss function, if a node with a few nonzero samples should be predicted as zero rather than very slightly positive. $\endgroup$
    – Ben Reiniger
    Commented Mar 31, 2020 at 2:38
  • $\begingroup$ @BenReiniger the problem I am encountering is that the model returns a constant value as a prediction (i.e.,, majority class, converging to the mean) rather than a range of predicted values which are close to zero. I have MSE as the loss function. $\endgroup$ Commented Mar 31, 2020 at 3:38
  • $\begingroup$ Oh, a constant is surprising. What do you mean by majority class here though? $\endgroup$
    – Ben Reiniger
    Commented Mar 31, 2020 at 16:24
  • $\begingroup$ By majority class I am referring to the majority of zero outcome values. $\endgroup$ Commented Mar 31, 2020 at 18:58
  • $\begingroup$ You tell the model that most of the outcomes are zero, and the model tends to make predictions that are close to zero. This sounds like correct behavior. What am I missing? $\endgroup$
    – Dave
    Commented Oct 11, 2023 at 16:49

1 Answer 1

2
$\begingroup$

I have been dealing with exactly the same situation but with even more rare non-zero events from marketing conversions. I have a few tips, but I don't feel I have really settled the best practices so I look forward to other people adding their observations! For the record I'm using Catboost. From what I have seen:

  1. Be extra careful about overfitting - cross validate and try low learning rates.
  2. You can't use Tweedie loss but do include MAE vs. RMSE as one of the parameters you test in your cross validation. MAE can reduce the emphasis on outliers and improve out of sample performance (sometimes, so cross validate!) I have even tried cross validations over most of the exotic evaluation metrics supported by Catboost but have found only MAE or RMSE to be best.
  3. Expect most predicted values to be on a smaller scale than the real values. The predicted values incorporate both the probability of a non-zero outcome and the expected magnitude, so it makes sense they tend to be close to zero. The problem is that RMSE or MAE evaluation metrics become very hard to interpret. My advice is to use r-squared to get a better intuition for the quality of your model fit.
  4. Although most values will tend to be close to zero be aware that gradient boosting regressions can produce predictions outside the range of your observed outcomes.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.