1
$\begingroup$

I am using Keras to classify images. I am following the Keras blog. The accuracy from predict_generator is not matching with the accuracy obtained from the confusion matrix, which I am computing using the scikit-learn package. I have included the relevant snippet of the code below

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Convolution2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense
import numpy as np
import theano
from sklearn.metrics import classification_report, confusion_matrix
y_actual = np.ones((nb_test_samples),dtype = int)
y_actual[0:2817] = 0

train_datagen = ImageDataGenerator(
             featurewise_std_normalization=False,
             samplewise_std_normalization=False,
     rescale = 1./255)

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
                    train_data_dir,
                    target_size = (img_width,img_height),
                    batch_size = 32,
                    class_mode = 'binary')

test_generator = test_datagen.flow_from_directory(
            test_data_dir,
                    target_size = (img_width,img_height),
                    batch_size = 32,
            class_mode = 'binary',
            shuffle = False         )

model.fit_generator(
                    train_generator,
                    samples_per_epoch = nb_train_samples,
                    nb_epoch = nb_epoch,
                    validation_data = test_generator,
                    nb_val_samples = nb_test_samples)

score =   model.evaluate_generator(
                      test_generator,
                      4938)
print "Test fraction correct (Accuracy) = {:.2f}".format(score[1])
prediction = model.predict_generator(test_generator,nb_test_samples)

for i in xrange(0,len(prediction)):
    if prediction[i]<0.5:
       prediction[i] = 0
    else:
       prediction[i] = 1

#y_predicted = test_generator.classes
print np.sum(prediction)
CM = confusion_matrix(y_actual,prediction)
print CM 

If I use y_predicted, I get a perfectly diagonal confusion matrix, when the console output shows an accuracy of 70% which doesn't make any sense at all. What is that I am doing wrong?

$\endgroup$
  • $\begingroup$ Same problem. Did you fix this? $\endgroup$ – user24516 Sep 20 '16 at 19:18
  • $\begingroup$ @YimingYan I haven't been able to fix this. Have you been able to make any progress into this? $\endgroup$ – Raghuram Sep 22 '16 at 17:52
  • $\begingroup$ Did you fixed it ? Kindly explain how you did. $\endgroup$ – WaterRocket8236 Nov 9 '17 at 10:41
1
$\begingroup$

The score method used in keras does not calculate accuracy like the sklearn's accuracy_score method. Check the internal implementation of evaluate method to understand more. If you want to calculate accuracy, I suggest you to use sklearn's accuracy_score by getting the predictions or manually calculate if it is easier for you.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks for pointing out the difference. Just so that I have understood it correctly, in this link, blog.keras.io/…, accuracies of ~91% are reported using VGG net. So, this might not be 91% or 90% or whatever that Keras reports? $\endgroup$ – Raghuram Oct 13 '16 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.