Evaluating a Multi-Label Classification model

Question

I currently have a multi-label classification problem, for which I am using keras to build a neural network as follows:

n_cols = dataset.shape[1]
print(n_cols)

model = Sequential()
model.add(Dense(128, activation='relu', input_shape=(n_cols,)))
model.add(Dense(64, activation='relu'))
model.add(Dense(26, activation='sigmoid')) # Sigmoid for multi-label classification

sgd = SGD(lr=0.1, decay=1e-6, momentum=0.5, nesterov=True)
#RMSprop
model.compile(loss='binary_crossentropy', optimizer='RMSprop', metrics=['accuracy'])

model.summary()

## Fit the model ##
early_stopping_monitor = EarlyStopping(patience=20)
history = model.fit(dataset, labels, validation_split=0.33, epochs=30, callbacks=[early_stopping_monitor])

plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
plt.show()

I was informed that for multi-label classification, we use binary_crossentropy for the loss while having sigmoid for activation in the final layer (output layer). However, with this I am getting a resulting accuracy and val_accuracy of ~0.0931 and ~0.0937 respectively.

For the multi-label classification, is using the accuracy metric the best fit? I've looked around and some suggest that other metrics such as binary_accuracy may be better..

So the question is, how can one best evaluate the multi-label classification?

EDIT: For reference, I have 26 label columns in my target "classes" and the dataset consist of 21 columns. The entire dataset the model is trained on has ~82k samples.

Answer 1

You are correct for using sigmoid+binary CE for multi-label classification problem. On the other hand, try to think about how accuracy would be defined for this problem? I would use categorical accuracy!

Answer 2

If you choose metrics=['accuracy'], Keras automatically infers the accuracy metric according to the loss function. Four your case, since the loss function is BinaryCrossentropy, Keras has already chosen the metrics=['BinaryAccuracy'].

Answer 3

Binary crossentropy is for two class problem. You must use sparse categorical crossentropy for your multiclass classification problem and softmax in the last layer not sigmoid. Sigmoid and binary crissentropy is for two class problem while sparse categorical crossentropy and softmax is for multiclass problem.