What is a good method to generate the KS-statistic in R?

Question

Kolmogorov-Smirnov (KS) statistics is one of the commonly used measures to assess predictive power for marketing or credit risk models.

The KS statistic is usually published for logistic regression problems to give an indication of the quality of the model.

A Wikipedia page (https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) gives this explanation:

The KS statistic calculation below is from page 5 on https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf

require(ROCR)
set.seed(7)
prd=runif(1000)
act=round(prd)
prd[sample(1000,500)]=runif(500) #noise

pred<-prediction(prd,act)
perf <- performance(pred,"tpr","fpr")
#this code builds on ROCR library by taking the max delt
#between cumulative bad and good rates being plotted by
#ROCR see https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf
ks=max(attr(perf,'y.values')[[1]]-attr(perf,'x.values')[[1]])
plot(perf,main=paste0(' KS=',round(ks*100,1),'%'))
lines(x = c(0,1),y=c(0,1))
print(ks);
auc <- performance(pred, measure = "auc")
auc <- [email protected][[1]]
print(auc)

It gives a KS of 0.4939511 and an AUC of 0.7398465. Area under the curve is obviously correct as one can see.

Is the R code for the KS statistic correct? If not, what should it be?

Update

Warning: From a sample decision tree output which is very clumpy, the Sharma method gives a better estimate of the KS score while the ks.test gives a very bad one.

     auc         ks.score.D^+ sharma.ks
    0.6153846    0.7045455    0.2307692 

The ks.test function gives this error: 1: In ks.test(prd, act, alternative = "greater") : cannot compute exact p-value with ties

act=c(1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 
  0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 
  0, 0, 1)
prd=c(0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.352941176470588, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.914893617021277, 0.914893617021277, 0.914893617021277, 
  0.914893617021277, 0.352941176470588, 0.352941176470588, 0.914893617021277, 
  0.914893617021277) ##this only has two values
ks.test(prd, act, alternative='greater')$statistic
#gives 0.7045455 which clearly isn't correct

Answer 1

Also consider using the builtin ks.test:

> ks.test(prd, act, alternative='greater')

        Two-sample Kolmogorov-Smirnov test

data:  prd and act
D^+ = 0.494, p-value < 2.2204e-16
alternative hypothesis: the CDF of x lies above that of y

Answer 2

Yes, this is the correct way to use the KS statistic to evaluate the performance of a model. As the true positive and false positive rates are already cumulative density functions over the range 0:1, they reasonably can fit into the paradigm of the more general use of the K-S statistic as it applies to comparison of these types of functions.