Cost-sensitive Logloss for XGBoost

Question

I want to use the following asymmetric cost-sensitive custom logloss objective function, which has an aversion for false negatives simply by penalizing them more, with XGBoost.

$$ \begin{array} \\ p &= \frac{1}{1+e^{-x}} \\ \hat{y} &= min(max(p, 10^{-7}, 1-10^{-7}) \\ FN &= y \times log(\hat{y}) \\ FP &= (1-y) \times log(1-\hat{y}) \\ Loss &= \frac{-1}{N}\sum_i 5 \times FN + FP \end{array} $$ I have calculated the gradient and hessian for this loss function: $$ \begin{array} \\ \frac{dLoss}{dx} &= 4py + p - 5y \\ \frac{d^2Loss}{dx^2} &= (4y + 1) * p (1.0 - p) \end{array} $$

And my code:

def logistic_obj(y_hat, dtrain):
    y = dtrain.get_label()
    p = 1.0 / (1.0 + np.exp(-y_hat))
    grad = 4 * p * y + p - 5 * y
    hess = (4 * y + 1) * (p * (1.0 - p))
    return grad, hess

def err_rate(y_hat, dtrain):
    y = dtrain.get_label()
    y_hat = np.clip(y_hat, 10e-7, 1-10e-7)
    loss_fn = y*np.log(y_hat)
    loss_fp = (1.0 - y)*np.log(1.0 - y_hat)
    return 'error', np.sum(-(5*loss_fn+loss_fp))/len(y)

xgb_pars = {'eta': 0.2, 'objective': 'binary:logistic', 
      'max_depth': 6, 'tree_method': 'hist', 'seed': 42}

model_trn = xgb.train(xgb_pars, d_trn, 10, evals=[(d_trn, 'trn'), 
            (d_val, 'vld')], obj=logistic_obj, feval=err_rate)

Running the code in verbose mode prints out the following. The two columns on the right hand side gives the error calculated by my own error calculation function passed as feval. I'm not sure why XGBoost still shows the error calculated by its own objective, but the problem is that it apparently hasn not used my updating rules, as its error decreases but my custom error starts increasing after five iterations. If I comment out the objective directive, it apparently defaults to RMSE which makes matters worse.

[0] trn-error:0.065108  vld-error:0.056749  trn-error:0.782048  vld-error:0.755389
[1] trn-error:0.064876  vld-error:0.056645  trn-error:0.727871  vld-error:0.695685
[2] trn-error:0.064487  vld-error:0.05651   trn-error:0.699920  vld-error:0.662203
[3] trn-error:0.064573  vld-error:0.056553  trn-error:0.691798  vld-error:0.64864
[4] trn-error:0.064484  vld-error:0.056514  trn-error:0.698498  vld-error:0.649974
[5] trn-error:0.064483  vld-error:0.056514  trn-error:0.716450  vld-error:0.662659
[6] trn-error:0.064470  vld-error:0.056507  trn-error:0.742848  vld-error:0.683847
[7] trn-error:0.064466  vld-error:0.056506  trn-error:0.775665  vld-error:0.71153
[8] trn-error:0.064435  vld-error:0.056497  trn-error:0.813440  vld-error:0.744165
[9] trn-error:0.064164  vld-error:0.056393  trn-error:0.854973  vld-error:0.780628

Answer 1

I'm answering my question. Well, I've ditched XGBoost for LightGBM (the only Microsoft product I seem to enjoy by far) but since the interface is quite similar, the same should apply to XGBoost. Apparently I don't need to apply Sigmoid to predictions. I don't know why the examples suggest otherwise.

def logistic_obj(y_hat, dtrain):
    y = dtrain.get_label()
    p = y_hat # p = 1. / (1. + np.exp(-y_hat))
    grad = p - y
    hess = p * (1. - p)
    grad = 4 * p * y + p - 5 * y
    hess = (4 * y + 1) * (p * (1.0 - p))
    return grad, hess

def err_rate(y_hat, dtrain):
    y = dtrain.get_label()
    # y_hat = 1.0 / (1.0 + np.exp(-y_hat))
    y_hat = np.clip(y_hat, 10e-7, 1-10e-7)
    loss_fn = y*np.log(y_hat)
    loss_fp = (1.0 - y)*np.log(1.0 - y_hat)
    return 'error', np.sum(-(5*loss_fn+loss_fp))/len(y), False

Answer 2

For those of you interested in penalising false negatives false positives, the derivation is as follows:

Let the loss for the $i$th instance be $L$. As usual, $p(x)$ is the sigmoid function. Finally, we will put a weight $\beta >1$ on the false positives.

$$L = -y\ln p - \beta(1-y) \ln(1-p)$$

$$\text{grad} = \frac{\partial L}{\partial x} = \frac{\partial L}{\partial p} \frac{\partial p}{\partial x} = p(\beta + y -\beta y) -y$$

$$\text{hess} = \frac{\partial^2 L}{\partial x^2} = p(1-p)(\beta + y -\beta y)$$

The code is then

def weighted_logloss(y_hat, dtrain):
    y = dtrain.get_label()
    p = y_hat
    beta = 5
    grad = p * (beta + y - beta*y) - y
    hess = p * (1 - p) * (beta + y - beta*y)
    return grad, hess