2
$\begingroup$

I am using this code for a multilabel problem classification.

from __future__ import print_function

from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Embedding
from keras.layers import LSTM
from keras.models import Model
from keras.datasets import imdb

max_features = len(vocabDic)
maxlen = 500  # cut texts after this number of words (among top max_features most common words)
batch_size = 32

train_set = sequence.pad_sequences(train_set, maxlen=maxlen)
test_set = sequence.pad_sequences(test_set, maxlen=maxlen)

print('train_set shape:', train_set.shape)
print('test_set shape:', test_set.shape)

print('Build model...')
model = Sequential()
model.add(Embedding(max_features, 128))
model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(90, activation='sigmoid'))

# try using different optimizers and different optimizer configs
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

print('Train...')
model.fit(train_set, train_labels,
          batch_size=batch_size,
          epochs=15,
          validation_data=(test_set, test_labels))
score, acc = model.evaluate(test_set, test_labels,
                            batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)

My problem is there are 90 classes and the accuracy is too high from the second epoch. I suspect keras is computing something incorrectly. Any clues?

Edit: I calculated the total recall of the model. It is barely 5%( 5 times better than random). Is it a normal behavior for such a problem?

Improve this question
$\endgroup$
3
  • $\begingroup$ It may happen and is not strange. depending on your data you may have even better results in first epoch. I've seen the similar thing in mnist $\endgroup$ – Media Dec 17 '17 at 21:21
  • $\begingroup$ I see you're using binary cross-entropy for your cost function. For multi-class classification you could look into categorical cross-entropy and categorical accuracy for your loss and metric, and troubleshoot with sklearn.metrics.classification_report on your test set $\endgroup$ – redhqs Dec 18 '17 at 11:07
  • $\begingroup$ Just to clarify: are you talking about multi-label (individual samples may belong to more than one classes) or multi-class (individual samples belong to one and only one class) classification here? $\endgroup$ – desertnaut Dec 19 '17 at 16:20
3
$\begingroup$

Why are you using binary_crossentropy? You should be using categorical_crossentropy. However, if you insist on using binary_crossentropy change your metric to metrics=['binary_accuracy', 'categorical_accuracy'] (this will display both accuracies).

Improve this answer
$\endgroup$
3
  • 6
    $\begingroup$ Using binary_crossentropy is in fact correct. If the problem is a multi-label classification problem, it turns into K binary classification problems. Using softmax would be wrong, as doing that would result in raising the probability on one class lowering others. $\endgroup$ – Oliver Ni Nov 18 '18 at 2:11
  • 1
    $\begingroup$ This comment by @OliverNi is correct. $\endgroup$ – cgnorthcutt Jan 21 '19 at 6:22
  • $\begingroup$ The answer has 3 upvotes and at the same time, a comment which is pointing a mistake @Tophat should update the answer or clarify the comment. Especially this point "Why are you using binary_crossentropy? You should be using categorical_crossentropy" $\endgroup$ – 10xAI Jun 19 '20 at 7:17
2
$\begingroup$

I have answered a similar question here.

Your problem is that Accuracy is not the right metric for multi-label tasks. Try something different like AUC, precision, recall, accuracy@k, precision@recall. The choice of binary_crossentropy is correct since you are predicting each label independently.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.