I have a small time series dataset of about 3000 samples and 5 features. With xgboost, my predictions seem biased (consistently overestimating the target). No matter how many estimators I throw at the problem along with hyperparameter tuning, I can't seem to beat a random forest. How can I go about diagnosing this?
rf = RandomForestRegressor(n_estimators=1000,
max_features=1,
min_samples_leaf=20,
random_state=0, n_jobs=-1)
xgb = xgb.XGBRegressor(n_estimators=100000,
max_depth=1,
learning_rate=.0001,
min_child_weight=1,
subsample=.1,
colsample_bytree=.1,
base_score=0)