-1
$\begingroup$

I'm training a XGBoost regressor in Python on a data set with a large number of indicator variables (one-hot-encoded from categorical variables) and a few numerical variables.The dataset size is over a million rows with a total column number of ~1000. The parameter I used was:

param = {
    'objective':            'reg:linear',
    'bst:max_depth':        6, 
    "min_child_weight":     1,
    'gamma':                5,
    'max_delta_step':       1,
    'bst:eta':              0.01,
    'nthread':              16,
    'verbose':              1
}
num_rounds = 1000

I checked the param of fitted trees and seems like most of them have a depth of 1 or 2. This happened for this feature size of ~1000; for a smaller feature size, the prediction result seems OK (but theoretically ~1000 would provide a better prediction result, so that's why I want it to work).

enter image description here

Does this mean that there's only very few variables are used in the tree construction and results in most of the trees being identical? Or am I doing anything very stupid?

The dataset is quite large and I'm not sure how to provide a runnable sample. Any possible suggestions or general discussions are welcomed!

$\endgroup$
0
$\begingroup$

Did you check the separation gain of individual variables? It might be that most of them have no separation power. This could result in the same prediction for everything.

$\endgroup$
  • $\begingroup$ What is a good way to check something like this? It might be the case as most of the response is 0 or very low and the assumption of normality is not quite met. $\endgroup$ – Yilun Zhang May 8 '17 at 18:20
  • $\begingroup$ I would have a look at the correlation between variables and the regressed value. Additionally, you can (if your data makes this possible) split your data into different parts dependent of the regressed value (essentially binning and then classify the bins). For these classes you could then use a Kolgomorov Smirnov Test on the input variables and see if the underlying distributions are different. $\endgroup$ – El Burro May 9 '17 at 6:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.