I want to use the following asymmetric cost-sensitive custom logloss objective function, which has an aversion for false negatives simply by penalizing them more, with XGBoost.
$$ \begin{array} \\ p &= \frac{1}{1+e^{-x}} \\ \hat{y} &= min(max(p, 10^{-7}, 1-10^{-7}) \\ FN &= y \times log(\hat{y}) \\ FP &= (1-y) \times log(1-\hat{y}) \\ Loss &= \frac{-1}{N}\sum_i 5 \times FN + FP \end{array} $$ I have calculated the gradient and hessian for this loss function: $$ \begin{array} \\ \frac{dLoss}{dx} &= 4py + p - 5y \\ \frac{d^2Loss}{dx^2} &= (4y + 1) * p (1.0 - p) \end{array} $$
And my code:
def logistic_obj(y_hat, dtrain):
y = dtrain.get_label()
p = 1.0 / (1.0 + np.exp(-y_hat))
grad = 4 * p * y + p - 5 * y
hess = (4 * y + 1) * (p * (1.0 - p))
return grad, hess
def err_rate(y_hat, dtrain):
y = dtrain.get_label()
y_hat = np.clip(y_hat, 10e-7, 1-10e-7)
loss_fn = y*np.log(y_hat)
loss_fp = (1.0 - y)*np.log(1.0 - y_hat)
return 'error', np.sum(-(5*loss_fn+loss_fp))/len(y)
xgb_pars = {'eta': 0.2, 'objective': 'binary:logistic',
'max_depth': 6, 'tree_method': 'hist', 'seed': 42}
model_trn = xgb.train(xgb_pars, d_trn, 10, evals=[(d_trn, 'trn'),
(d_val, 'vld')], obj=logistic_obj, feval=err_rate)
Running the code in verbose mode prints out the following. The two columns on the right hand side gives the error calculated by my own error calculation function passed as feval
. I'm not sure why XGBoost still shows the error calculated by its own objective, but the problem is that it apparently hasn not used my updating rules, as its error decreases but my custom error starts increasing after five iterations. If I comment out the objective
directive, it apparently defaults to RMSE which makes matters worse.
[0] trn-error:0.065108 vld-error:0.056749 trn-error:0.782048 vld-error:0.755389
[1] trn-error:0.064876 vld-error:0.056645 trn-error:0.727871 vld-error:0.695685
[2] trn-error:0.064487 vld-error:0.05651 trn-error:0.699920 vld-error:0.662203
[3] trn-error:0.064573 vld-error:0.056553 trn-error:0.691798 vld-error:0.64864
[4] trn-error:0.064484 vld-error:0.056514 trn-error:0.698498 vld-error:0.649974
[5] trn-error:0.064483 vld-error:0.056514 trn-error:0.716450 vld-error:0.662659
[6] trn-error:0.064470 vld-error:0.056507 trn-error:0.742848 vld-error:0.683847
[7] trn-error:0.064466 vld-error:0.056506 trn-error:0.775665 vld-error:0.71153
[8] trn-error:0.064435 vld-error:0.056497 trn-error:0.813440 vld-error:0.744165
[9] trn-error:0.064164 vld-error:0.056393 trn-error:0.854973 vld-error:0.780628