I have a small time series dataset of about 3000 samples and 5 features. With xgboost, my predictions seem biased (consistently overestimating the target). No matter how many estimators I throw at the problem along with hyperparameter tuning, I can't seem to beat a random forest. How can I go about diagnosing this?
rf = RandomForestRegressor(n_estimators=1000, max_features=1, min_samples_leaf=20, random_state=0, n_jobs=-1) xgb = xgb.XGBRegressor(n_estimators=100000, max_depth=1, learning_rate=.0001, min_child_weight=1, subsample=.1, colsample_bytree=.1, base_score=0)