How to use "tree boosting" with a data-driven loss function

Question

We have a problem which has a data-driven (non-analytical) loss function. Our target contains whole numbers between 0 and 20 (the target is inherently discrete), although larger values are possible, just not present in our dataset. The fact that we have a very precise loss function leaves us with some serious issues when using algorithms like XGBoost:

The loss function is generally non-convex. It's not easily fitted by a convex function since its shape is data-driven and can vary drastically. For example, this means that a large punishment is inevitably given for predictions further from the part of the function that is well-fitted, where no large punishment is required. If we interpolate instead of fit, the hessian can be negative (see attached picture), which is a problem for determining leaf weights (right?).

We think we can adapt something like the XGBoost algorithm (I use this algorithm as an example because I'm both familiar with the paper and the API) by swapping out its dependance on the gradient en hessian with a brute-force method for finding the optimal leaf weights and best gain. However, this will slow down the algorithm massively, perhaps cripplingly so.

My questions are: is the some default way of dealing with complex loss-functions within existing algorithms? Is the an algorithm that is suited for dealing with these problems? Is there anything else you could suggest to solve the above issues?

Thanks in advance.

Answer 1

First some prior and known aclarations (that you probably already know).

Metric is what we want to optimize.

Optimization Loss is what the model optimizes.

Obviously, we would like the Metric and the optimization loss to be the same, but this is always not possible. How to deal with this?

• Run the right model. Some models can optimize different loss functions. In the case of XGBoost you have two loss functions, the one of the decision tree and the one of the boosting.

• Preprocess the target and optimize another metric, this will be for example transforming the target to the logarithmic of the target and then in that space applying a known loss function

• Optimize another loss function and metric and then post-process the predictions.

• Write your own cost functions. For xgboost we implement a single function that takes predictions and target values and computes the first and second-order derivatives.

• Optimize another metric and use early stopping.

The last one almost always works.

In general for complex algorithms Neural Networks tend to work better due to the flexibility of the loss functions (more than in normal ML).

Answer 2

With XGBoost you can come up with your own loss and metric. It is relatively simple to just add a custom loss. However, I have no experiance with problems described by you, so you would need to see if what you have in mind will fit into the standard XGB.

Find an implementation of custom loss (R) here: https://github.com/Bixi81/R-ml/blob/master/xgboost_custom_objective_fair_loss.R

Answer 3

As for me: I don’t see any problem except for the simple logics of XGBoosting:

With weak learner – you’re getting 1st derivative (gradient).

Further with strong lerner – you’re getting 2nd derivative (hessian). – that crossing Zero-mark illustrates the extremum of your 1st derivative. – that is the purpose of gradient-descent (find minimum of parabolic shape).

As so as “ XG Boost splits up to the maximum depth specified and prunes the tree backward and removes splits beyond which there is an only negative loss.” – it identifies these extremums in stronger learner (that serves to make more accurate predictions than previously weak-learner did). XGBoost have no problems seeing “negative values” – it uses them for the purpose.

And XGBoost is doing it in parallel (comparing with other bosting algorithms – see link above) – why do you consider the algorithm to be the cause of “slow down” ?

The main disadvantage of XGBoost, I know, that it works poorly with highly-dimensional data. But this problem is common for most algos, therefore dimensionality-reduction in advance is worthwhile anyway.

The only problem in your logics I see in your figures (if I understood you correctly)– is taking 2nd derivative from loss function. But in Algorithm’s strong-lerner the derivative is taken from losses taken from weak learner (whose aim is to classify with either losses putting further these losses to stronger-lerner) => Thus from losses themselves (residuals) you need only one derivative got in stronger-lerner, as to my opinion, as to Algorithm description at the link I've left.

Yes, and I agree with previous answer, that problem could be either in your choice of loss function(s) (can make your own loss functions) or in your data preprocessing (either transformation for data preprocessing before modelling: 1. Logarithmic, 2. Square Root, 3. Inverse (1/x) -- according data nature), also possibly you could forget to scale data beforehand.

It is also hard to understand, why do you write about “interpolate instead of fit” – as so as interpolation is used in i.e. k-NN algorithm – aiming for clustering (as unsupervised method), but XGBoost aims to classification/regresson as supervised in its nature, if you didn’t change this algorithm by your own way somehow… I hesitate to comment more till will be sure you’re using Boosting algorithm for the proper purpose & in the right way.

P.S. In my 2-core 32x PC XGBoost still could not be build in Python – therefore just in Colab can use it (to feel its speed due to parallel execution on probably multi-core Colab’s host)