2
$\begingroup$

Kolmogorov-Smirnov (KS) statistics is one of the commonly used measures to assess predictive power for marketing or credit risk models.

The KS statistic is usually published for logistic regression problems to give an indication of the quality of the model.

A Wikipedia page (https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) gives this explanation:

KS statistic

The KS statistic calculation below is from page 5 on https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf

require(ROCR)
set.seed(7)
prd=runif(1000)
act=round(prd)
prd[sample(1000,500)]=runif(500) #noise

pred<-prediction(prd,act)
perf <- performance(pred,"tpr","fpr")
#this code builds on ROCR library by taking the max delt
#between cumulative bad and good rates being plotted by
#ROCR see https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf
ks=max(attr(perf,'y.values')[[1]]-attr(perf,'x.values')[[1]])
plot(perf,main=paste0(' KS=',round(ks*100,1),'%'))
lines(x = c(0,1),y=c(0,1))
print(ks);
auc <- performance(pred, measure = "auc")
auc <- auc@y.values[[1]]
print(auc)

It gives a KS of 0.4939511 and an AUC of 0.7398465. Area under the curve is obviously correct as one can see.

From R code

Is the R code for the KS statistic correct? If not, what should it be?

Update

Warning: From a sample decision tree output which is very clumpy, the Sharma method gives a better estimate of the KS score while the ks.test gives a very bad one.

     auc         ks.score.D^+ sharma.ks
    0.6153846    0.7045455    0.2307692 

The ks.test function gives this error: 1: In ks.test(prd, act, alternative = "greater") : cannot compute exact p-value with ties

Dec Tree

act=c(1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 
  0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 
  0, 0, 1)
prd=c(0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.352941176470588, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.352941176470588, 0.352941176470588, 0.914893617021277, 
  0.914893617021277) ##this only has two values
ks.test(prd, act, alternative='greater')$statistic
#gives 0.7045455 which clearly isn't correct
$\endgroup$
3
$\begingroup$

Also consider using the builtin ks.test:

> ks.test(prd, act, alternative='greater')

        Two-sample Kolmogorov-Smirnov test

data:  prd and act
D^+ = 0.494, p-value < 2.2204e-16
alternative hypothesis: the CDF of x lies above that of y
$\endgroup$
2
  • $\begingroup$ I tried ks.test before but could not get it to work. The trick is alternative='greater'. That is what I was looking for. $\endgroup$ – Chris Jun 8 '17 at 21:33
  • $\begingroup$ Oh my, sometimes ks.test gives a KS score that is very wrong when it gives this message: In ks.test(prd, act, alternative = "greater") : cannot compute exact p-value with ties $\endgroup$ – Chris Jun 24 '17 at 13:03
2
$\begingroup$

Yes, this is the correct way to use the KS statistic to evaluate the performance of a model. As the true positive and false positive rates are already cumulative density functions over the range 0:1, they reasonably can fit into the paradigm of the more general use of the K-S statistic as it applies to comparison of these types of functions.

enter image description here

$\endgroup$
2
  • $\begingroup$ I confirmed that the above graph difference and the ROC chart do generate the same KS score. $\endgroup$ – Chris Jun 8 '17 at 21:33
  • $\begingroup$ Would you please provide the code to replicate your plot? $\endgroup$ – majom Aug 30 '17 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.