0
$\begingroup$

I have been trying to implement logistic regression in python. Basically the code works and it gives the accuracy of the predictive model at a level of 91% but for some reason the AUC score is 0.5 which is basically the worst possible score because it means that the model is completely random. Also the classification report returns error: "UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. 'precision', 'predicted', average, warn_for)". Does anyone know what should I change so it works properly?

 import numpy as np
 import pandas as pd
 from sklearn.cross_validation import train_test_split
 from sklearn.linear_model import LogisticRegression
 from sklearn.metrics import accuracy_score  
 from sklearn.preprocessing import StandardScaler
 from sklearn.metrics import roc_auc_score
 from sklearn.metrics import classification_report

 data_file = pd.read_csv('loan.csv', delimiter=',')

 # variable preprocessing

 data_file['loan_status'] = np.where(data_file['loan_status'].isin(['Fully 
 Paid', 'Current']), 1, 0)
 loan_stat=data_file['loan_status']
 loan_stat=loan_stat.astype(np.float64)

 m = {
    'n/a': 0,     
    '< 1 year': 0,
    '1 year': 1,
    '2 years': 2,
    '3 years': 3,
    '4 years': 4,
    '5 years': 5,
    '6 years': 6,
    '7 years': 7,
    '8 years': 8,
    '9 years': 9,
    '10+ years': 10
 }
 emp_length=data_file.emp_length.map(m)
 emp_length.astype(np.float64)

 annual_inc=data_file['annual_inc']
 delinq_2yrs=data_file['delinq_2yrs']
 dti=data_file['dti']
 loan_amnt=data_file['loan_amnt']
 installment=data_file['installment']
 int_rate=data_file['int_rate']
 total_acc=data_file['total_acc']
 open_acc=data_file['open_acc']
 pub_rec=data_file['pub_rec']
 acc_now_delinq=data_file['acc_now_delinq']

 #variables combined into one dataframe

 X=pd.DataFrame()

 X['annua_inc']=annual_inc
 X['delinq_2yrs']=delinq_2yrs
 X['dti']=dti
 X['emp_length']=emp_length
 X['loan_amnt']=loan_amnt
 X['installment']=installment
 X['int_rate']=int_rate
 X['total_acc']=total_acc
 X['open_acc']=open_acc
 X['pub_rec']=pub_rec
 X['acc_now_delinq']=acc_now_delinq
 X['loan_stat']=loan_stat

 X=X.dropna(axis=0)
 y=X['loan_stat']
 X=X.drop(['loan_stat'], axis=1)

 scaler=StandardScaler()
 X=scaler.fit_transform(X)

 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, 
 random_state=42)

 model=LogisticRegression(penalty='l2', C=1)
 model.fit(X_train, y_train)
 score=accuracy_score(y_test, model.predict(X_test))
 roc=roc_auc_score(y_test, model.predict(X_test))
 cr=classification_report(y_test, model.predict(X_test))

Here is the link to the data: https://www.kaggle.com/wendykan/lending-club-loan-data/downloads/lending-club-loan-data.zip

$\endgroup$
4
$\begingroup$

In order to calculate the AUC, you need to have probabilities. Therefore you should use the following function:

roc=roc_auc_score(y_test, model.predict_proba(X_test)[:,1])

This will give you the probability for each sample in X_test having label 1.

$\endgroup$
  • $\begingroup$ It worked. thanks. but the classification report still returns the above mentioned error. Do you know why this may be? $\endgroup$ – Blazej Kowalski Jun 4 '17 at 15:14
  • $\begingroup$ Not sure, what does your y look like? Might be due to this if you're only predicting one class: stackoverflow.com/questions/34757653/… $\endgroup$ – Archie Jun 4 '17 at 15:20
  • $\begingroup$ my y is a vector of "1" and "0" $\endgroup$ – Blazej Kowalski Jun 4 '17 at 15:21
  • $\begingroup$ And your predictions, also both 1 and 0? $\endgroup$ – Archie Jun 4 '17 at 15:22
  • $\begingroup$ unfortunately predictions give me 1 for all samples. So does it mean that the report is correct? but the there are 600k samples in the set. it is very unlikely that all predictions would be 1 $\endgroup$ – Blazej Kowalski Jun 4 '17 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.