I am using XGBoost for a classification problem. The model has multiple inputs and a probability as output.

For simplicity's sake, let's say the model only has one continous input and the function to be learned is f(x) = x.

I am creating training samples for it using the f(x) = x relationship. Of course, in reality, I cannot create an endless amount of training samples and am restricted by the given data set.

The model output would look something like this:

enter image description here

As can be seen, the model predictions are noisy. As the x variable goes from 0 to 1, the y variable jitters around the true relationship. I would want to reduce / penalize this jitter. For every delta x, I want to to keep the delta y as small as possible but as high as necessary. Are there any model parameters to do this? Of course, I can just smooth the predictions ex post but I'd like to tell it to the model directly to make the smoothness vs. optimal prediction trade-off.

  • $\begingroup$ You can use regularization, such as lambda in XGBoost, although you will still not get a diagonal line but a step function (if using trees). $\endgroup$ Nov 15, 2023 at 12:11
  • $\begingroup$ I have tried this (sweeped all sensible regularization parameter combinations). Unfortunately, the effect is a 1-1 trade-off with out-of-sample prediction accuracy and therefore not feasible. $\endgroup$
    – Neo
    Nov 15, 2023 at 15:45
  • $\begingroup$ In that case maybe the relationship is not as smooth as you think. If the relationship is truly smooth have you tried using a linear model with appropriate terms? $\endgroup$ Nov 15, 2023 at 15:49
  • $\begingroup$ I am quite sure it is smooth but not linear. A neural network creates a smoother prediction, however, I would prefer a tree based model if possible. $\endgroup$
    – Neo
    Nov 15, 2023 at 15:55
  • $\begingroup$ Trees cannot create smooth functions, it's a feature. If you want smooth you need to look elsewhere. If you know the nature of the relationship, say quadratic, you could add this variable to the model and it might help. $\endgroup$ Nov 15, 2023 at 16:00


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.