0
$\begingroup$

I have made a neural network that was working correctly as a multi-class classifier, but after changing the loss and the activation function, plus the output layer to just 1 neuron, it is not working correctly. I hope you can tell me what is wrong with my code:

    # Define the MLP model
model = Sequential()
model.add(Dense(X_train.shape[1], input_dim= X_train.shape[1], activation= 'sigmoid'))
model.add(Dense(2*X_train.shape[1], activation= 'sigmoid'))
model.add(Dense(2*X_train.shape[1],activation= 'sigmoid'))
model.add(Dense(1,activation= 'sigmoid', kernel_initializer='normal'))
model.compile(loss='binary_crossentropy', optimizer='Adam', metrics=['accuracy'])
monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, 
                        verbose=1, mode='auto', restore_best_weights=True)


# Train the model
model.fit(X_train, Y_train,validation_data=(X_test,Y_test),callbacks=[monitor], epochs=10, batch_size=64, verbose=1)

# Evaluate the model on the test set
y_pred = model.predict(X_test)
y_pred_classes = y_pred

y_preds = np.argmax(y_pred_classes, axis=1)

# Calculate and print accuracy
accuracy = accuracy_score(Y_test, y_preds)
print('Accuracy of MLP: ' + str(accuracy))

# Calculate and print precision, recall, and F1-score
precision, recall, fscore, _ = precision_recall_fscore_support(Y_test, y_preds, average='weighted')
print('Precision of MLP: ' + str(precision))
print('Recall of MLP: ' + str(recall))
print('F1-score of MLP: ' + str(fscore))

# Print the classification report and confusion matrix
print(classification_report(Y_test, y_preds,target_names=target_names))

import matplotlib.pyplot as plt
import seaborn as sns
cm=confusion_matrix(Y_test,y_preds)
f,ax=plt.subplots(figsize=(10,10))
sns.heatmap(cm,annot=True,cmap='Blues',linewidth=0.5,fmt=".0f",ax=ax,xticklabels=target_names, yticklabels=target_names)
plt.xlabel("Predicted")
plt.ylabel("Real")
plt.show()

enter image description here

Thanks for your help.

$\endgroup$
4
  • $\begingroup$ You validation accuracy is high. There are many potential factors involved, e.g. leakage of the training data into the validation data that makes the validation accuracy artificially high, different distribution of the validation and test sets (e.g. different class imbalances), etc. $\endgroup$
    – noe
    Nov 10, 2023 at 10:11
  • $\begingroup$ Hello noe. The problem is that the network used worked correctly doing a multi-class classification. I have just changed the last neuron, the activation function of the last layer and the loss function. As you can see in the code I am using X_train and X_test, which I have split after preprocessing and oversampling. Also, in the confussion matrix, you can see, obviously a 2x2 matrix, and the results for 'Attack' class is 0. I don't know what is happening. $\endgroup$ Nov 10, 2023 at 18:15
  • $\begingroup$ Why should it work? Perhaps it would help for you to explain this explanation. $\endgroup$
    – Dave
    Nov 10, 2023 at 21:15
  • $\begingroup$ It has been working correctly for multi-class classifiaction, i guess it shouldn't have any problem working as a binary classifier. I have tried different architectures for an MLP and all have a 0.9929 accuracy and around .9860 precision. I also tried with an LSTM network and the result is the same. $\endgroup$ Nov 11, 2023 at 10:46

1 Answer 1

2
$\begingroup$

I think the problem is here

y_pred = model.predict(X_test)
y_pred_classes = y_pred
y_preds = np.argmax(y_pred_classes, axis=1)

Your model outputs a single value (Dense(1,activation= 'sigmoid', kernel_initializer='normal'))

If n_test is the number of items in X_test, your output y_pred will be of size (n_test, 1).

When you call y_preds = np.argmax(y_pred_classes, axis=1), you argmax along a unit dimension, so you just get a vector of 0s.

You need to instead set a threshold and determine predictions by y_pred_classes = y_pred > threshold

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.