I am very new to machine learning and I am trying to use XGBoostRegressor for my machine learning model (it has to do with physical modeling).

I found out that it works very well for predicting the behavior of the quantity I try to predict (flux), but it underestimates the amplitude of change. From the 2D histogram I derived that the predicted flux equals 0.76*observed + constant offset.

What can create such bias in the model? Does it have to do with metric being rmse? How can I improve it? (I also tried LightGBM and have the same problem).

# my XGBoost code (quite standard)
est = XGBRegressor(n_estimators=500, max_depth=11, subsample=0.8, colsample_bytree=0.8, 
                   learning_rate=0.01, early_stopping_rounds=10, n_jobs=-1, random_state=0)
est.fit(input_train, output_train)

And also

# my LightGBM code:
params = {
    'boosting_type': 'gbdt',
    'n_estimators': 5000,
    'objective': 'regression',
    'metric': {'l1','l2'},
    'colsample_by_tree': 0.8,
    'learning_rate': 0.05,
    'max_depth': 5,
    'min_data_in_leaf': 500,
    'reg_alpha': 0.63,
    'reg_lambda': 0.35,  
    'feature_fraction': 0.8,
    'bagging_fraction': 0.8,
    'bagging_freq': 5,
    'verbose': 0
lgb_train = lgb.Dataset(input_train, output_train)
lgb_eval = lgb.Dataset(input_eval, output_eval, reference=lgb_train)
gbm = lgb.train(params,lgb_train, 
                valid_sets=lgb_eval, early_stopping_rounds=500)

I attach the pictures. What can be causing this? enter image description here

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.