3
$\begingroup$

I want to use the following asymmetric cost-sensitive custom logloss objective function, which has an aversion for false negatives simply by penalizing them more, with XGBoost.

$$ \begin{array} \\ p &= \frac{1}{1+e^{-x}} \\ \hat{y} &= min(max(p, 10^{-7}, 1-10^{-7}) \\ FN &= y \times log(\hat{y}) \\ FP &= (1-y) \times log(1-\hat{y}) \\ Loss &= \frac{-1}{N}\sum_i 5 \times FN + FP \end{array} $$ I have calculated the gradient and hessian for this loss function: $$ \begin{array} \\ \frac{dLoss}{dx} &= 4py + p - 5y \\ \frac{d^2Loss}{dx^2} &= (4y + 1) * p (1.0 - p) \end{array} $$

And my code:

def logistic_obj(y_hat, dtrain):
    y = dtrain.get_label()
    p = 1.0 / (1.0 + np.exp(-y_hat))
    grad = 4 * p * y + p - 5 * y
    hess = (4 * y + 1) * (p * (1.0 - p))
    return grad, hess

def err_rate(y_hat, dtrain):
    y = dtrain.get_label()
    y_hat = np.clip(y_hat, 10e-7, 1-10e-7)
    loss_fn = y*np.log(y_hat)
    loss_fp = (1.0 - y)*np.log(1.0 - y_hat)
    return 'error', np.sum(-(5*loss_fn+loss_fp))/len(y)

xgb_pars = {'eta': 0.2, 'objective': 'binary:logistic', 
      'max_depth': 6, 'tree_method': 'hist', 'seed': 42}

model_trn = xgb.train(xgb_pars, d_trn, 10, evals=[(d_trn, 'trn'), 
            (d_val, 'vld')], obj=logistic_obj, feval=err_rate)

Running the code in verbose mode prints out the following. The two columns on the right hand side gives the error calculated by my own error calculation function passed as feval. I'm not sure why XGBoost still shows the error calculated by its own objective, but the problem is that it apparently hasn not used my updating rules, as its error decreases but my custom error starts increasing after five iterations. If I comment out the objective directive, it apparently defaults to RMSE which makes matters worse.

[0] trn-error:0.065108  vld-error:0.056749  trn-error:0.782048  vld-error:0.755389
[1] trn-error:0.064876  vld-error:0.056645  trn-error:0.727871  vld-error:0.695685
[2] trn-error:0.064487  vld-error:0.05651   trn-error:0.699920  vld-error:0.662203
[3] trn-error:0.064573  vld-error:0.056553  trn-error:0.691798  vld-error:0.64864
[4] trn-error:0.064484  vld-error:0.056514  trn-error:0.698498  vld-error:0.649974
[5] trn-error:0.064483  vld-error:0.056514  trn-error:0.716450  vld-error:0.662659
[6] trn-error:0.064470  vld-error:0.056507  trn-error:0.742848  vld-error:0.683847
[7] trn-error:0.064466  vld-error:0.056506  trn-error:0.775665  vld-error:0.71153
[8] trn-error:0.064435  vld-error:0.056497  trn-error:0.813440  vld-error:0.744165
[9] trn-error:0.064164  vld-error:0.056393  trn-error:0.854973  vld-error:0.780628
$\endgroup$
2
$\begingroup$

I'm answering my question. Well, I've ditched XGBoost for LightGBM (the only Microsoft product I seem to enjoy by far) but since the interface is quite similar, the same should apply to XGBoost. Apparently I don't need to apply Sigmoid to predictions. I don't know why the examples suggest otherwise.

def logistic_obj(y_hat, dtrain):
    y = dtrain.get_label()
    p = y_hat # p = 1. / (1. + np.exp(-y_hat))
    grad = p - y
    hess = p * (1. - p)
    grad = 4 * p * y + p - 5 * y
    hess = (4 * y + 1) * (p * (1.0 - p))
    return grad, hess

def err_rate(y_hat, dtrain):
    y = dtrain.get_label()
    # y_hat = 1.0 / (1.0 + np.exp(-y_hat))
    y_hat = np.clip(y_hat, 10e-7, 1-10e-7)
    loss_fn = y*np.log(y_hat)
    loss_fp = (1.0 - y)*np.log(1.0 - y_hat)
    return 'error', np.sum(-(5*loss_fn+loss_fp))/len(y), False
$\endgroup$
  • 2
    $\begingroup$ It's well and khub that you answer your question :) would you (mishe) explain second function. I don't understand the last line. $\endgroup$ – Media Jan 25 '18 at 15:45
  • $\begingroup$ Sure. err_rate is my implementation of the evaluation function. While it doesn't take part in optimization, this is what I'm trying to optimize, and it can be used for early stopping to prevent overfitting. It should return a triplet, the first simply being the name of the metric. The second is the value of our metric. And the last value which is a boolean indicates whether we're trying to maximize or minimize this metric. Apparently this last boolean value is only needed for LGBM. $\endgroup$ – Nima Mohammadi Jan 25 '18 at 21:10
  • $\begingroup$ Thanks and mamnun :), the string was a bit misleading, I don't remember why! $\endgroup$ – Media Jan 25 '18 at 21:24
  • 1
    $\begingroup$ @Media Chakerim ;) $\endgroup$ – Nima Mohammadi Jan 25 '18 at 22:40
1
$\begingroup$

For those of you interested in penalising false negatives false positives, the derivation is as follows:

Let the loss for the $i$th instance be $L$. As usual, $p(x)$ is the sigmoid function. Finally, we will put a weight $\beta >1$ on the false positives.

$$L = -y\ln p - \beta(1-y) \ln(1-p)$$

$$\text{grad} = \frac{\partial L}{\partial x} = \frac{\partial L}{\partial p} \frac{\partial p}{\partial x} = p(\beta + y -\beta y) -y$$

$$\text{hess} = \frac{\partial^2 L}{\partial x^2} = p(1-p)(\beta + y -\beta y)$$

The code is then

def weighted_logloss(y_hat, dtrain):
    y = dtrain.get_label()
    p = y_hat
    beta = 5
    grad = p * (beta + y - beta*y) - y
    hess = p * (1 - p) * (beta + y - beta*y)
    return grad, hess
$\endgroup$
  • $\begingroup$ I've checked my calculations again and they seem to be correct. Notice that I've used $\beta$ to penalize FNs but you've used $\beta$ on FPs. If we set $\beta=1$ both of our gradients reduce to $p-y$, suggesting that they are both probably correct (except yours penalizes false positives for $\beta > 1$) $\endgroup$ – Nima Mohammadi Feb 3 '18 at 11:16
  • 1
    $\begingroup$ @NimaMohammadi My sincere apologies. You are, of course, correct. I have amended the answer slightly, but will leave it up in case anybody wants the explicit solution to penalise false positives. $\endgroup$ – surelyourejoking Feb 3 '18 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.