2
$\begingroup$

I have an issue with xgboost custom objectives: I do not manage to get consistent forecasts. In other words, the scale of my forecasts is not in line with the values I would like to predict. I tried many custom loss, but I always get the same issue.

import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.datasets import make_regression

n_samples_train = 500
n_samples_test = 100
n_features = 200

X, y = make_regression(n_samples_train, n_features,noise=10)
X_test, y_test = make_regression(n_samples_test, n_features,noise=10)

param = {'verbosity' : 1,
      'max_depth' : 12,
      'learning_rate' : 0.01,
      'nthread' : 3,
        }

dtrain = xgb.DMatrix(X, y)

best_nrounds = 50

bst_reglinear = xgb.train(param, 
                      dtrain, 
                      best_nrounds)


def reg_obj(preds,dtrain):
    y = dtrain.get_label()
    N = len(y)
    #residual = (preds-y).astype("float")
    grad = 2*preds-y
    hess = 2*N*np.ones(len(y))
    return grad, hess

bst_custom = xgb.train(param,
                   dtrain,
                   best_nrounds,
                   obj = reg_obj)



dtest = xgb.DMatrix(X_test)


pred = bst_reglinear.predict(dtest)
print(np.abs(pred).mean())

pred_custom = bst_custom.predict(dtest)
print(np.abs(pred_custom).mean())
$\endgroup$
1
  • $\begingroup$ Compare default xgboost objective function implementation with what you implemented, if they diverge your scores could be diverging $\endgroup$
    – Noah Weber
    Commented Dec 17, 2019 at 11:59

1 Answer 1

0
$\begingroup$

I think your grad and hess go wrong, try:

grad = 2*(preds-y)
hess = 2*np.ones(len(y))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.