2
$\begingroup$

I have developed the following model for a Binary Classifier. I have to evluate using roc_auc_score. I am getting unusual values for roc_auc_score.

When I use
epoch = 1, roc_auc_score = 0.8
epoch = 2, roc_auc_score =0.53

And after this for any epoch it is constant at 0.5. Why is this happening? Is there something wrong with my model?

DATA PREPROCESSING

#Load Dataset
test = pd.read_csv('test.csv')
train = pd.read_csv('train.csv')

#Combine Train and Test set for Data Cleaning
train['set'] = 'train'
test['set'] = 'test'
df = pd.concat([test, train])

#One Hot Encoding
df = pd.get_dummies(df, columns=['Gender','Driving_License','Previously_Insured','Vehicle_Age','Vehicle_Damage'])

#Moving Target Column to End
target = df['Response']
df.drop(labels=['Response'], axis=1, inplace = True)
df.insert(16, 'Response', target)

#Separating Train and Test Data
train = df[df['set']=='train']
test = df[df['set']=='test']
train = train.drop('set', 1)
test = test.drop('set', 1)

#Creating Input Features and Target Variables
X= train.iloc[:,1:15]
y= train.iloc[:,[15]]

#Standardizing the Input Features
scaler = StandardScaler()
X = scaler.fit_transform(X)

#Train Test Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

CODE

#Model
model = Sequential()

model.add(Dense(14, activation='relu', kernel_initializer='random_normal', input_dim=14))
  
#Output Layer
model.add(Dense(1, activation = 'sigmoid', kernel_initializer='random_normal'))

#Compiling the neural network
model.compile(optimizer ='adam',loss='binary_crossentropy', metrics =['accuracy'])

#Fitting the data to the training dataset  
model.fit(X_train,y_train, batch_size=32, epochs=1, verbose=0) 

#Make predictions and convert to binary value
pred_train = model.predict(X_train)
pred_test = model.predict(X_test)  

#ROC AUC Score
print('Train AUC = {:.5f}'.format(roc_auc_score(y_train,pred_train)))
print('Test AUC = {:.5f}'.format(roc_auc_score(y_test,pred_test)))

#Accuracy
print('Train Accuracy = {:.3f}'.format(accuracy_score(y_train,pred_train.round())))
print('Test Accuracy = {:.3f}'.format(accuracy_score(y_test,pred_test.round())))
$\endgroup$
9
  • 1
    $\begingroup$ Are those roc_auc_scores measured on train or test? If it's train, it's very strange, cos the code looks fine. Maybe you've preprocessed you data somehow incorrectly. $\endgroup$ Sep 13, 2020 at 17:32
  • $\begingroup$ roc_auc_score is same for both datasets form epoch=2. Should I add my data preprocessing? $\endgroup$ Sep 13, 2020 at 17:34
  • $\begingroup$ Well, the problem maybe somewhere there. If that code isn't difficult to read, I can glance $\endgroup$ Sep 13, 2020 at 17:39
  • $\begingroup$ @MichaelSolotky I have added the data preprocessing, it's fairly simple. First I have combined the train and test set, then performed one hot encoding on required features and then again separated train and test set. Just to mention performing Data Standardization also gave the same result. $\endgroup$ Sep 13, 2020 at 17:50
  • $\begingroup$ Why only 1 layer. Have you tried with 5-6 layers $\endgroup$
    – 10xAI
    Sep 13, 2020 at 18:14

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.