Find Net Reclassification Improvement/Index metric using Python

Question

I am working on a binary classification problem with ~5k records and class proportion of 33:67.

I have 60 features/variables in my dataset and finally I have come to about 10 variables based on multiple feature selection algorithms and domain understanding.

Now I would like to experiment with this 10 features and get the best of these 10 which gives the high performance.

For example, Having 6 features might give an AUC of 84% whereas adding one new feature (7 features) might give an AUC of 85%.

But adding two features (8 features), might give an AUC of 83%.

I read in several posts and F Harrel blog posts that AUC is not always a good metric to compare model performance.

q1) So what are the other best metrics that you usually use?

q2) How to calculate Net Reclassification Improvement? Read it's better than AUC.

Can you people please help me?

Answer 1

Just stick to F1.

Yes NRI is cools but there are edgecases and its not a magic bullet. Read this paper

IF you really want to implement it you can embedd r programs in python.

Since there is already implementation in r what you could do is something like this. Just change it to use NRI.