I've been exploring the use of XGBoost in many different applications. Up to now, I always find the best results with shallow trees (from 1 to 3 levels), with the rest of the parameters very dependent on the problem.

On my current assignment, I found that I get a much better performance if I use >300 trees, with a depth >20 !! I understand that this is saying a lot about the complexity of the issue, but I can stop wondering if there is some feature transformation I could do to change this. EX: by doing PCA and adding the resulting component to a dataset, one can sometimes not only replace several features by a smaller number but also capture higher level relashionships

  • $\begingroup$ have you tried to add more estimators? Even in thw order of thousands of trees? It should mean less overfitting $\endgroup$
    – German C M
    Commented Feb 20, 2021 at 15:54
  • $\begingroup$ I did hyperparameter exploration using a Bayesian method. However, my point is that independently of the exploration method, the set of hyperparameters that minimise logloss always contain high depth trees. There is no evident overfitting. I was trying to discuss/ask was whether one can create some features that capture this complexity and create a simpler model ( and what would be preferable and why) $\endgroup$ Commented Feb 21, 2021 at 16:57


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.