1
$\begingroup$

I have a small time series dataset of about 3000 samples and 5 features. With xgboost, my predictions seem biased (consistently overestimating the target). No matter how many estimators I throw at the problem along with hyperparameter tuning, I can't seem to beat a random forest. How can I go about diagnosing this?

rf = RandomForestRegressor(n_estimators=1000, 
                      max_features=1, 
                      min_samples_leaf=20,
                      random_state=0, n_jobs=-1)

xgb = xgb.XGBRegressor(n_estimators=100000,
                             max_depth=1,
                             learning_rate=.0001,
                             min_child_weight=1, 
                             subsample=.1, 
                             colsample_bytree=.1,
                             base_score=0)

enter image description here

|improve this question
$\endgroup$
  • 2
    $\begingroup$ Isn’t it possible random forest is just the best algorithm for your data? $\endgroup$ – astel Sep 22 at 3:47
  • $\begingroup$ There's probably nothing to diagnose, your mistake was to assume that decision trees are too old for competing with the young generation ;) $\endgroup$ – Erwan Sep 23 at 1:25
  • $\begingroup$ I just thought any problem RF can do, XGB can do better! Kidding aside, it's frustrating to not know why certain algos fit certain data better. $\endgroup$ – kliao Sep 23 at 21:37
  • $\begingroup$ en.wikipedia.org/wiki/No_free_lunch_theorem $\endgroup$ – Erwan Sep 24 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.