Comparing XGBR with CatBoost performance

Question

I saw on a CatBoost site that it supposed to outperform any other boosted training model and decided to try it myself on a Kaggle's https://www.kaggle.com/c/house-prices-advanced-regression-techniques.

I created some basic kernel without any complex preprocessing, feature selection, GridSearch, stacking, etc... just to compare the XGBR and CatBoost performance. But as far as I see the XGBR always outperforms the CatBoost.
https://www.kaggle.com/markbquant/compare-catboostregressor-vs-xgbregressor

The CatBoost parameters: iterations=100, depth=3, learning_rate=0.1
The XGBR parameters: subsample=0.7, colsample_bytree=0.7, n_estimators=500, learning_rate=0.03, max_depth=5, min_child_weight=3

For example, inside the Kaggle, the XGBR received score 0.134 and the CatBoot only reached 0.197 (I tried both one_hot_max_size and cat_features). I'll be thankful if someone could point me what is wrong with the CatBoost model, may be some optimization is missing.

Answer 1

Performing such benchmark is not that easy. Meaning one can not just pick a few data set and run these models as there is a data dependency. In such cases, one need to simulate data through various process - the simulation helps to design various data in various condition. for example perhaps model one is doing a better job at binning so, the data with various binning condition must be in place before hand, or the depth of the tree. So just picking house-prices data is not enough.

Bare in mind such out performance can be really really small. Don't expect 10% difference ! they lay down within 1% often.

What distinguish Catboost from Xgboost is the thread safing in the production environment. Xgboost is not a thread safe - therefore, can not be used in any serious deployment environment.