I have developed the following model for a Binary Classifier. I have to evluate using roc_auc_score
. I am getting unusual values for roc_auc_score
.
When I use
epoch = 1, roc_auc_score = 0.8
epoch = 2, roc_auc_score =0.53
And after this for any epoch it is constant at 0.5. Why is this happening? Is there something wrong with my model?
DATA PREPROCESSING
#Load Dataset
test = pd.read_csv('test.csv')
train = pd.read_csv('train.csv')
#Combine Train and Test set for Data Cleaning
train['set'] = 'train'
test['set'] = 'test'
df = pd.concat([test, train])
#One Hot Encoding
df = pd.get_dummies(df, columns=['Gender','Driving_License','Previously_Insured','Vehicle_Age','Vehicle_Damage'])
#Moving Target Column to End
target = df['Response']
df.drop(labels=['Response'], axis=1, inplace = True)
df.insert(16, 'Response', target)
#Separating Train and Test Data
train = df[df['set']=='train']
test = df[df['set']=='test']
train = train.drop('set', 1)
test = test.drop('set', 1)
#Creating Input Features and Target Variables
X= train.iloc[:,1:15]
y= train.iloc[:,[15]]
#Standardizing the Input Features
scaler = StandardScaler()
X = scaler.fit_transform(X)
#Train Test Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
CODE
#Model
model = Sequential()
model.add(Dense(14, activation='relu', kernel_initializer='random_normal', input_dim=14))
#Output Layer
model.add(Dense(1, activation = 'sigmoid', kernel_initializer='random_normal'))
#Compiling the neural network
model.compile(optimizer ='adam',loss='binary_crossentropy', metrics =['accuracy'])
#Fitting the data to the training dataset
model.fit(X_train,y_train, batch_size=32, epochs=1, verbose=0)
#Make predictions and convert to binary value
pred_train = model.predict(X_train)
pred_test = model.predict(X_test)
#ROC AUC Score
print('Train AUC = {:.5f}'.format(roc_auc_score(y_train,pred_train)))
print('Test AUC = {:.5f}'.format(roc_auc_score(y_test,pred_test)))
#Accuracy
print('Train Accuracy = {:.3f}'.format(accuracy_score(y_train,pred_train.round())))
print('Test Accuracy = {:.3f}'.format(accuracy_score(y_test,pred_test.round())))