Making cpp function from xgboost dump_model() output

Question

I'd like to use the output of an xgboost BDT model in a code base without having to rely explicitly on xgboost or otherwise. Using a modified version of this script, xgb2cpp I am able to generate a cpp function which, to my eye, takes exactly the output of a trained network dumped to a txt file and converts to a cpp function. However, when running this function with the exact same inputs, I cannot replicate probabilities output by xgboost itself.

Here's a minimal working example of the code used to make the BDT:

import xgboost as xgb
import numpy as np

# Create dummy data for training and testing
X_train = np.random.rand(100, 4)
y_train = np.random.randint(0, 2, size=100)

X_test = np.random.rand(20, 4)

clf = xgb.XGBClassifier(n_estimators=2, min_child_weight=5, gamma=0.5, max_depth=2)
clf.fit(X_train, y_train)
y_pred_proba = clf.predict_proba(X_test)

# dump model to txt file
clf.get_booster().dump_model('test_minimal/dump.raw.txt')

print(X_test)
print(y_pred_proba)

The dumped model looks like this:

booster[0]:
0:[f0<0.460618258] yes=1,no=2,missing=2
    1:leaf=-0.10681767
    2:[f2<0.684740245] yes=3,no=4,missing=4
        3:leaf=0.000418715936
        4:leaf=0.176613688
booster[1]:
0:[f3<0.255429506] yes=1,no=2,missing=2
    1:leaf=-0.136423871
    2:[f0<0.607831895] yes=3,no=4,missing=4
        3:leaf=-0.00219070958
        4:leaf=0.125992939

with corresponding cpp:

float classify(std::vector<float> &sample) {

    float sum = 0.0;

    if (sample[0] <0.460618258) {
        sum += -0.10681767;
    } else {
        if (sample[2] <0.684740245) {
            sum += 0.000418715936;
        } else {
            sum += 0.176613688;
        }
    }


    if (sample[3] <0.255429506) {
        sum += -0.136423871;
    } else {
        if (sample[0] <0.607831895) {
            sum += -0.00219070958;
        } else {
            sum += 0.125992939;
        }
    }


    return sum;
}

When testing for probabilities I pass the output of the cpp function to the sigmoid function to convert to a probability:

float sigmoid(float x) {
    return 1.0 / (1.0 + exp(-x));
}

But this result does not match the result from xbgoost... so what gives? Does xgboost do something I am not realizing? I checked out their code and it seems it does exactly what I've done. Others have pointed out that a starting value of 0.5 is used by default, so I even accounted for this:

float sigmoid(float x) {
    return 1.0 / (1.0 + exp(-x + 0.5));
}

Either I'm missing something or this isn't currently possible. Any help is appreciated!

Answer 1

I'm not sure entirely why this works... but if I explicitly set base_score = 0.5 in xgb.XGBClassifier then everything works. As it turns out, I don't need to add/subtract 0.5 from the output either. So I run all the code as above and then converted the score to a probability using

float sigmoid(float x) {
    return 1.0 / (1.0 + exp(-x));
}

and the results matched!