AUC and classification report in Logistic regression in python

Question

I have been trying to implement logistic regression in python. Basically the code works and it gives the accuracy of the predictive model at a level of 91% but for some reason the AUC score is 0.5 which is basically the worst possible score because it means that the model is completely random. Also the classification report returns error: "UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. 'precision', 'predicted', average, warn_for)". Does anyone know what should I change so it works properly?

 import numpy as np
 import pandas as pd
 from sklearn.cross_validation import train_test_split
 from sklearn.linear_model import LogisticRegression
 from sklearn.metrics import accuracy_score  
 from sklearn.preprocessing import StandardScaler
 from sklearn.metrics import roc_auc_score
 from sklearn.metrics import classification_report

 data_file = pd.read_csv('loan.csv', delimiter=',')

 # variable preprocessing

 data_file['loan_status'] = np.where(data_file['loan_status'].isin(['Fully 
 Paid', 'Current']), 1, 0)
 loan_stat=data_file['loan_status']
 loan_stat=loan_stat.astype(np.float64)

 m = {
    'n/a': 0,     
    '< 1 year': 0,
    '1 year': 1,
    '2 years': 2,
    '3 years': 3,
    '4 years': 4,
    '5 years': 5,
    '6 years': 6,
    '7 years': 7,
    '8 years': 8,
    '9 years': 9,
    '10+ years': 10
 }
 emp_length=data_file.emp_length.map(m)
 emp_length.astype(np.float64)

 annual_inc=data_file['annual_inc']
 delinq_2yrs=data_file['delinq_2yrs']
 dti=data_file['dti']
 loan_amnt=data_file['loan_amnt']
 installment=data_file['installment']
 int_rate=data_file['int_rate']
 total_acc=data_file['total_acc']
 open_acc=data_file['open_acc']
 pub_rec=data_file['pub_rec']
 acc_now_delinq=data_file['acc_now_delinq']

 #variables combined into one dataframe

 X=pd.DataFrame()

 X['annua_inc']=annual_inc
 X['delinq_2yrs']=delinq_2yrs
 X['dti']=dti
 X['emp_length']=emp_length
 X['loan_amnt']=loan_amnt
 X['installment']=installment
 X['int_rate']=int_rate
 X['total_acc']=total_acc
 X['open_acc']=open_acc
 X['pub_rec']=pub_rec
 X['acc_now_delinq']=acc_now_delinq
 X['loan_stat']=loan_stat

 X=X.dropna(axis=0)
 y=X['loan_stat']
 X=X.drop(['loan_stat'], axis=1)

 scaler=StandardScaler()
 X=scaler.fit_transform(X)

 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, 
 random_state=42)

 model=LogisticRegression(penalty='l2', C=1)
 model.fit(X_train, y_train)
 score=accuracy_score(y_test, model.predict(X_test))
 roc=roc_auc_score(y_test, model.predict(X_test))
 cr=classification_report(y_test, model.predict(X_test))

Here is the link to the data: https://www.kaggle.com/wendykan/lending-club-loan-data/downloads/lending-club-loan-data.zip

Answer 1

In order to calculate the AUC, you need to have probabilities. Therefore you should use the following function:

roc=roc_auc_score(y_test, model.predict_proba(X_test)[:,1])

This will give you the probability for each sample in X_test having label 1.