XGBoost increase the error when changing evaluation function

Question

I have changed the eval function of XGBoost to rmsle and the optimisation increase the error after the iteration [2] instead of decreasing it. If I change to the default eval function, RMSE, this does not happen.

This is the code of RMSLE used:

def evalerror(preds, dtrain): # this is compatible with DMatrix
    labels = dtrain.get_label()
    assert len(preds) == len(labels)
    labels = labels.tolist()
    preds = preds.tolist()
    terms_to_sum = [(math.log(labels[i] + 1) - math.log(max(0,preds[i]) + 1)) ** 2.0 for i,pred in enumerate(labels)]
    return 'error', (sum(terms_to_sum) * (1.0/len(preds))) ** 0.5

This is the parameters of XGBoost used:

param = {'bst:max_depth':1, 'bst:eta':0.025, 'silent':False, 'objective':'reg:linear','eval_metric':'rmse' }
bst = xgb.train( param, d_train, num_rounds,early_stopping_rounds=20, evals=eval_list, verbose_eval=True, feval=evalerror)

and this is the evaluation:

[0] eval-error:0.836219 train-error:0.835095
Multiple eval metrics have been passed: 'train-error' will be used for early stopping.

Will train until train-error hasn't improved in 20 rounds.
[1] eval-error:0.809301 train-error:0.806747
[2] eval-error:0.792647 train-error:0.78908
[3] eval-error:0.803355 train-error:0.798805
[4] eval-error:0.803261 train-error:0.79835
[5] eval-error:0.809352 train-error:0.804283
[6] eval-error:0.810453 train-error:0.805126
[7] eval-error:0.811059 train-error:0.805646
[8] eval-error:0.815261 train-error:0.809722
[9] eval-error:0.820237 train-error:0.814521
[10]    eval-error:0.823378 train-error:0.817408
[11]    eval-error:0.824981 train-error:0.81868
[12]    eval-error:0.826607 train-error:0.820176
[13]    eval-error:0.827813 train-error:0.821358
[14]    eval-error:0.827625 train-error:0.821007
[15]    eval-error:0.823347 train-error:0.816547
[16]    eval-error:0.824362 train-error:0.81752
[17]    eval-error:0.82529  train-error:0.818321
[18]    eval-error:0.824621 train-error:0.817463
[19]    eval-error:0.824103 train-error:0.816766
[20]    eval-error:0.814759 train-error:0.807234
[21]    eval-error:0.807961 train-error:0.800186
[22]    eval-error:0.808398 train-error:0.800246
Stopping. Best iteration:
[2] eval-error:0.792647 train-error:0.78908

It may be the case that I need to adjust my objective function to this evaluation metric?

Answer 1

If your goal is to minimize the RMSLE, the easier way is to transform the labels directly into log scale and use reg:linear as objective (which is the default) and rmse as evaluation metric. This way XGBoost will be minimizing the RMSLE direclty. You can achieve this by setting:

dtrain = DMatrix(X_train, label=np.log1p(y_train))

where np.log1p(x) is equal to np.log(x+1).

When you want to make your prediction in the original space, you will need to compute the inverse transform of np.log1p, that is to say np.expm1:

predictions = np.expm1(bst.predict(dtest))

If you are just interested into monitoring the RMSLE through the training of your XGBoost which actually is minimizing the RMSE, then you should expect to see the RMSLE behave a little strangely as it is not what you are minimizing.

Answer 2

While using RMSLE, you should pay attention to the point that the metric penalises the under predicted values more than the over predicted values, hence driving the the model towards high bias. This might be changing the weights of a model and hence increasing error after second iteration. you can go to the following link