After running xgboost model with:

objective = 'binary:logistic'
eval_metric = 'logloss' 

I have a group of 3 variables that have the highest values of gain. Now, if I replace each one of the 20 more important variables according to this metric by their mean one by one and calculate the kolmogorov smirnov coefficient (KS), I get that the one that reduces the most the ks is not one of those 3, but one that has a relative low gain.


    Gain    Cover
v1  21.5%   2.5%
v2  12.9%   4.1%
v3  11.1%   1.8%
v4  3.5%    3.4%
v5  2.7%    1.7%
v6  2.4%    2.5%
v7  2.3%    2.2%
v8  2.2%    1.9%
v9  1.9%    4.0%
v10 1.9%    2.0%
v11 1.9%    0.9%
v12 1.6%    4.6% *****

ks of replacing each variable by its mean (one by one)

v12         39% *****
the rest    45%

How is this explained? Thanks.

  • 1
    $\begingroup$ Could you add some detail about which two set of values did you perform the KS test on? $\endgroup$ – user12075 Sep 19 '18 at 3:50
  • $\begingroup$ @user12075 what kind of details do you think it will be necessary? $\endgroup$ – GabyLP Sep 26 '18 at 14:43
  • $\begingroup$ If you used two sample KS test, then which two set of values did you perform the KS test on? If you used one sample KS test, what values and what distribution did you test? $\endgroup$ – user12075 Sep 26 '18 at 15:48
  • $\begingroup$ @user12075, the 2 samples are the scores for the positives vs the scores for the negatives... so my test would be the difference between those 2 distributions. It is a way to measure the performance of the model. $\endgroup$ – GabyLP Sep 27 '18 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.