0
$\begingroup$

I am very new to machine learning and I am trying to use XGBoostRegressor for my machine learning model (it has to do with physical modeling).

I found out that it works very well for predicting the behavior of the quantity I try to predict (flux), but it underestimates the amplitude of change. From the 2D histogram I derived that the predicted flux equals 0.76*observed + constant offset.

What can create such bias in the model? Does it have to do with metric being rmse? How can I improve it? (I also tried LightGBM and have the same problem).

# my XGBoost code (quite standard)
est = XGBRegressor(n_estimators=500, max_depth=11, subsample=0.8, colsample_bytree=0.8, 
                   learning_rate=0.01, early_stopping_rounds=10, n_jobs=-1, random_state=0)
est.fit(input_train, output_train)

And also

# my LightGBM code:
params = {
    'boosting_type': 'gbdt',
    'n_estimators': 5000,
    'objective': 'regression',
    'metric': {'l1','l2'},
    'colsample_by_tree': 0.8,
    'learning_rate': 0.05,
    'max_depth': 5,
    'min_data_in_leaf': 500,
    'reg_alpha': 0.63,
    'reg_lambda': 0.35,  
    'feature_fraction': 0.8,
    'bagging_fraction': 0.8,
    'bagging_freq': 5,
    'verbose': 0
}
lgb_train = lgb.Dataset(input_train, output_train)
lgb_eval = lgb.Dataset(input_eval, output_eval, reference=lgb_train)
gbm = lgb.train(params,lgb_train, 
                valid_sets=lgb_eval, early_stopping_rounds=500)

I attach the pictures. What can be causing this? enter image description here

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.