Based on the paper by Chen & Guestrin (2016) "XGBoost: A Scalable Tree Boosting System", XGBoost's "exact split finding algorithm enumerates over all the possible splits on all the features to find the best split" (page 3). Thus, my understanding was that XGBoost enumerates over all features and uses unique values of each feature as candidate split points to then choose the split value that maximizes the splitting criterion (gain).

Then, my question is why do the split values for float-type features chosen are often not one of the unique values for that feature? For example, for a certain feature in the data with floating point values, like so: 966.0, 1234.0, 2350.0, 4567.0 .... If Xgboost chooses to split on that feature, the split value can be, e.g. (feature < 1800.5), which is not one of the unique values for that feature, but is often the mid-point between two values.

Also, assuming no missing values in the data, for the ordinal type features, with the values like so: set(1, 2, 3, 4, 5, 6, 7, 8), declared as integers, how does XGBoost enumerate candidate split points? Does the algorithm treat that feature as categorical?

I've read a lot of good answers on the forum, like here and here, have gone thru xgboost documentation and tutorials, but they don't seem to elucidate on the specific procedure of enumerating candidate split points for exact greedy algorithm.

Could someone, perhaps, familiar with the source code could shed light on these questions?

  • $\begingroup$ @Winks would you mind answering this question? $\endgroup$ – tvl Aug 2 '18 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.