Difference in performance Sigmoid vs. Softmax

For the same Binary Image Classification task, if in the final layer I use 1 node with Sigmoid activation function and binary_crossentropy loss function, then the training process goes through pretty smoothly (92% accuracy after 3 epochs on validation data).

However, if I change the final layer to 2 nodes and use the Softmax activation function with sparse_categorical_crossentropy loss function, then the model doesn't seem to learn at all and stuck at 55% accuracy (the ratio of the negative class).

Is this difference in performance normal? I thought for a binary classification task, Sigmoid with Binary Crossentropy and Softmax with Sparse Categorical Crossentropy should output similar if not identical results? Or did I do something wrong?

Note: I use Adam optimizer and there is a single label column containing 0s and 1s.

Edit: Code for the 2 cases

Case 1: Sigmoid with binary_crossentropy

def addTopModelMobilNetV1(bottom_model, num_classes):
top_model = bottom_model.output
top_model = layers.GlobalAveragePooling2D()(top_model)
top_model = layers.Dense(1024, activation='relu')(top_model)
top_model = layers.Dense(1024, activation='relu')(top_model)
top_model = layers.Dense(512, activation='relu')(top_model)
top_model = layers.Dense(1, activation='sigmoid')(top_model)

# print(model.summary())

earlystopping_cb = callbacks.EarlyStopping(patience=3, restore_best_weights=True)
history = model.fit_generator(generator=train_generator,
steps_per_epoch=train_df.shape[0]//TRAIN_BATCH_SIZE,
validation_data = val_generator,
epochs = 10,
callbacks = [earlystopping_cb]
)


Case 2: Softmax with sparse_categorical_crossentropy

def addTopModelMobilNetV1(bottom_model, num_classes):
top_model = bottom_model.output
top_model = layers.GlobalAveragePooling2D()(top_model)
top_model = layers.Dense(1024, activation='relu')(top_model)
top_model = layers.Dense(1024, activation='relu')(top_model)
top_model = layers.Dense(512, activation='relu')(top_model)
top_model = layers.Dense(2, activation='softmax')(top_model)

earlystopping_cb = callbacks.EarlyStopping(patience=3, restore_best_weights=True)

history = model.fit_generator(generator=train_generator,
steps_per_epoch=train_df.shape[0]//TRAIN_BATCH_SIZE,
validation_data = val_generator,
epochs = 10,
callbacks = [earlystopping_cb]
)

• Why should these different activation functions give similar results? Jun 28, 2021 at 19:32
• I think you might read thoroughly the answers in this page. Believe me you will find the answer: https://stats.stackexchange.com/questions/233658/softmax-vs-sigmoid-function-in-logistic-classifier
– user119783
Jun 28, 2021 at 19:37
• @NikosM. Because the softmax function is an extension of sigmoid that works for any number of classes >= 2 and not just 2. For binary classification (2 classes), they are the same. Jun 28, 2021 at 19:46
• @Hamzah I checked out the link and it does confirm my confusion since for 2 classes softmax and sigmoid are identical. Did I use the softmax activation incorrectly somehow? Jun 28, 2021 at 19:47
• Can you elaborate how you get the predicted class when using 2 final nodes with softmax? Jun 29, 2021 at 8:24