# XGBoost increase the error when changing evaluation function

I have changed the eval function of XGBoost to rmsle and the optimisation increase the error after the iteration [2] instead of decreasing it. If I change to the default eval function, RMSE, this does not happen.

This is the code of RMSLE used:

def evalerror(preds, dtrain): # this is compatible with DMatrix
labels = dtrain.get_label()
assert len(preds) == len(labels)
labels = labels.tolist()
preds = preds.tolist()
terms_to_sum = [(math.log(labels[i] + 1) - math.log(max(0,preds[i]) + 1)) ** 2.0 for i,pred in enumerate(labels)]
return 'error', (sum(terms_to_sum) * (1.0/len(preds))) ** 0.5


This is the parameters of XGBoost used:

param = {'bst:max_depth':1, 'bst:eta':0.025, 'silent':False, 'objective':'reg:linear','eval_metric':'rmse' }
bst = xgb.train( param, d_train, num_rounds,early_stopping_rounds=20, evals=eval_list, verbose_eval=True, feval=evalerror)


and this is the evaluation:

[0] eval-error:0.836219 train-error:0.835095
Multiple eval metrics have been passed: 'train-error' will be used for early stopping.

Will train until train-error hasn't improved in 20 rounds.
[1] eval-error:0.809301 train-error:0.806747
[2] eval-error:0.792647 train-error:0.78908
[3] eval-error:0.803355 train-error:0.798805
[4] eval-error:0.803261 train-error:0.79835
[5] eval-error:0.809352 train-error:0.804283
[6] eval-error:0.810453 train-error:0.805126
[7] eval-error:0.811059 train-error:0.805646
[8] eval-error:0.815261 train-error:0.809722
[9] eval-error:0.820237 train-error:0.814521
[10]    eval-error:0.823378 train-error:0.817408
[11]    eval-error:0.824981 train-error:0.81868
[12]    eval-error:0.826607 train-error:0.820176
[13]    eval-error:0.827813 train-error:0.821358
[14]    eval-error:0.827625 train-error:0.821007
[15]    eval-error:0.823347 train-error:0.816547
[16]    eval-error:0.824362 train-error:0.81752
[17]    eval-error:0.82529  train-error:0.818321
[18]    eval-error:0.824621 train-error:0.817463
[19]    eval-error:0.824103 train-error:0.816766
[20]    eval-error:0.814759 train-error:0.807234
[21]    eval-error:0.807961 train-error:0.800186
[22]    eval-error:0.808398 train-error:0.800246
Stopping. Best iteration:
[2] eval-error:0.792647 train-error:0.78908


It may be the case that I need to adjust my objective function to this evaluation metric?

• xgboost tries to maximize the objective function. It sounds like you need to return negative RMSE if you want to minimize it. Aug 19, 2016 at 16:23
• I think is the opposite: it tries to minimize the objective function, which is the error... xgboost.readthedocs.io/en/latest/model.html Aug 19, 2016 at 16:27
• It is customizable. But you're right, at least in the python API maximize=False. Aug 19, 2016 at 16:35

If your goal is to minimize the RMSLE, the easier way is to transform the labels directly into log scale and use reg:linear as objective (which is the default) and rmse as evaluation metric. This way XGBoost will be minimizing the RMSLE direclty. You can achieve this by setting:

dtrain = DMatrix(X_train, label=np.log1p(y_train))


where np.log1p(x) is equal to np.log(x+1).

When you want to make your prediction in the original space, you will need to compute the inverse transform of np.log1p, that is to say np.expm1:

predictions = np.expm1(bst.predict(dtest))


If you are just interested into monitoring the RMSLE through the training of your XGBoost which actually is minimizing the RMSE, then you should expect to see the RMSLE behave a little strangely as it is not what you are minimizing.

• Thank you. I realized that it does not make sense to have an objective function different to the evaluation metric. Aug 20, 2016 at 0:09
• What if y_train contains value -1? np.log1p() will crash? Aug 28, 2017 at 11:20

While using RMSLE, you should pay attention to the point that the metric penalises the under predicted values more than the over predicted values, hence driving the the model towards high bias. This might be changing the weights of a model and hence increasing error after second iteration. you can go to the following link

• This is due to the nature of logarithmic function, let us suppose the correct prediction was 65 but the model underpredicted it to be 45 so |log(45) - log(65)| = 0.3677 (which is proportional to error) while if the model would over predict it to be 75 then |log(85) - log(65)| = 0.26826, we see that the error comes out to be more in case of under prediction, so, more penalty will be applied on a underprediction. PS: My language in last answer was a bit misleading, I have changed that now. Aug 20, 2016 at 11:04