2
$\begingroup$

I have changed the eval function of XGBoost to rmsle and the optimisation increase the error after the iteration [2] instead of decreasing it. If I change to the default eval function, RMSE, this does not happen.

This is the code of RMSLE used:

def evalerror(preds, dtrain): # this is compatible with DMatrix
    labels = dtrain.get_label()
    assert len(preds) == len(labels)
    labels = labels.tolist()
    preds = preds.tolist()
    terms_to_sum = [(math.log(labels[i] + 1) - math.log(max(0,preds[i]) + 1)) ** 2.0 for i,pred in enumerate(labels)]
    return 'error', (sum(terms_to_sum) * (1.0/len(preds))) ** 0.5

This is the parameters of XGBoost used:

param = {'bst:max_depth':1, 'bst:eta':0.025, 'silent':False, 'objective':'reg:linear','eval_metric':'rmse' }
bst = xgb.train( param, d_train, num_rounds,early_stopping_rounds=20, evals=eval_list, verbose_eval=True, feval=evalerror)

and this is the evaluation:

[0] eval-error:0.836219 train-error:0.835095
Multiple eval metrics have been passed: 'train-error' will be used for early stopping.

Will train until train-error hasn't improved in 20 rounds.
[1] eval-error:0.809301 train-error:0.806747
[2] eval-error:0.792647 train-error:0.78908
[3] eval-error:0.803355 train-error:0.798805
[4] eval-error:0.803261 train-error:0.79835
[5] eval-error:0.809352 train-error:0.804283
[6] eval-error:0.810453 train-error:0.805126
[7] eval-error:0.811059 train-error:0.805646
[8] eval-error:0.815261 train-error:0.809722
[9] eval-error:0.820237 train-error:0.814521
[10]    eval-error:0.823378 train-error:0.817408
[11]    eval-error:0.824981 train-error:0.81868
[12]    eval-error:0.826607 train-error:0.820176
[13]    eval-error:0.827813 train-error:0.821358
[14]    eval-error:0.827625 train-error:0.821007
[15]    eval-error:0.823347 train-error:0.816547
[16]    eval-error:0.824362 train-error:0.81752
[17]    eval-error:0.82529  train-error:0.818321
[18]    eval-error:0.824621 train-error:0.817463
[19]    eval-error:0.824103 train-error:0.816766
[20]    eval-error:0.814759 train-error:0.807234
[21]    eval-error:0.807961 train-error:0.800186
[22]    eval-error:0.808398 train-error:0.800246
Stopping. Best iteration:
[2] eval-error:0.792647 train-error:0.78908

It may be the case that I need to adjust my objective function to this evaluation metric?

$\endgroup$
  • $\begingroup$ xgboost tries to maximize the objective function. It sounds like you need to return negative RMSE if you want to minimize it. $\endgroup$ – Ricardo Cruz Aug 19 '16 at 16:23
  • $\begingroup$ I think is the opposite: it tries to minimize the objective function, which is the error... xgboost.readthedocs.io/en/latest/model.html $\endgroup$ – Escachator Aug 19 '16 at 16:27
  • 1
    $\begingroup$ It is customizable. But you're right, at least in the python API maximize=False. $\endgroup$ – Ricardo Cruz Aug 19 '16 at 16:35
5
$\begingroup$

If your goal is to minimize the RMSLE, the easier way is to transform the labels directly into log scale and use reg:linear as objective (which is the default) and rmse as evaluation metric. This way XGBoost will be minimizing the RMSLE direclty. You can achieve this by setting:

dtrain = DMatrix(X_train, label=np.log1p(y_train))

where np.log1p(x) is equal to np.log(x+1).

When you want to make your prediction in the original space, you will need to compute the inverse transform of np.log1p, that is to say np.expm1:

predictions = np.expm1(bst.predict(dtest))

If you are just interested into monitoring the RMSLE through the training of your XGBoost which actually is minimizing the RMSE, then you should expect to see the RMSLE behave a little strangely as it is not what you are minimizing.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thank you. I realized that it does not make sense to have an objective function different to the evaluation metric. $\endgroup$ – Escachator Aug 20 '16 at 0:09
  • $\begingroup$ What if y_train contains value -1? np.log1p() will crash? $\endgroup$ – harrison4 Aug 28 '17 at 11:20
3
$\begingroup$

While using RMSLE, you should pay attention to the point that the metric penalises the under predicted values more than the over predicted values, hence driving the the model towards high bias. This might be changing the weights of a model and hence increasing error after second iteration. you can go to the following link

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks Janpreet. Why is that? $\endgroup$ – Escachator Aug 20 '16 at 10:18
  • 2
    $\begingroup$ This is due to the nature of logarithmic function, let us suppose the correct prediction was 65 but the model underpredicted it to be 45 so |log(45) - log(65)| = 0.3677 (which is proportional to error) while if the model would over predict it to be 75 then |log(85) - log(65)| = 0.26826, we see that the error comes out to be more in case of under prediction, so, more penalty will be applied on a underprediction. PS: My language in last answer was a bit misleading, I have changed that now. $\endgroup$ – janpreet singh Aug 20 '16 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.