6
$\begingroup$

I am trying to build an image classification model with 2 classes with (1) or without (0). I can build the model and get an accuracy of 1. which is too good to be true (which is an issue) but when I use predict_generator as I have my images in folders, it only returns 1 class 0 (without class). There seems to be an issue but I can't work it out, i have looked at a number of articles but I still can't fix the issue.

image_shape = (220, 525, 3) #height, width, channels
img_width = 96
img_height = 96
channels = 3

epochs = 10

no_train_images = 11957              #!ls ../data/train/* | wc -l
no_test_images = 652                 #!ls ../data/test/* | wc -l
no_valid_images = 6156               #!ls ../data/test/* | wc -l

train_dir = '../data/train/'
test_dir = '../data/test/'
valid_dir = '../data/valid/'

classification_model = Sequential()

# First layer with 2D convolution (32 filters, (3, 3) kernel size 3x3, input_shape=(img_width, img_height, channels))
classification_model.add(Conv2D(32, (3, 3), input_shape=input_shape))
# Activation Function = ReLu increases the non-linearity
classification_model.add(Activation('relu'))
# Max-Pooling layer with the size of the grid 2x2
classification_model.add(MaxPooling2D(pool_size=(2, 2)))
# Randomly disconnets some nodes between this layer and the next 
classification_model.add(Dropout(0.2))

classification_model.add(Conv2D(32, (3, 3)))
classification_model.add(Activation('relu'))
classification_model.add(MaxPooling2D(pool_size=(2, 2)))
classification_model.add(Dropout(0.2))

classification_model.add(Conv2D(64, (3, 3)))
classification_model.add(Activation('relu'))
classification_model.add(MaxPooling2D(pool_size=(2, 2)))
classification_model.add(Dropout(0.25))

classification_model.add(Conv2D(64, (3, 3)))
classification_model.add(Activation('relu'))
classification_model.add(MaxPooling2D(pool_size=(2, 2)))
classification_model.add(Dropout(0.3))

classification_model.add(Flatten())
classification_model.add(Dense(64))
classification_model.add(Activation('relu'))
classification_model.add(Dropout(0.5))
classification_model.add(Dense(1))
classification_model.add(Activation('sigmoid'))

# Using binary_crossentropy as we only have 2 classes
classification_model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])



batch_size = 32

# this is the augmentation configuration we will use for training
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    zoom_range=0.2)

# this is the augmentation configuration we will use for testing:
# only rescaling
valid_datagen = ImageDataGenerator(rescale=1. / 255)
test_datagen = ImageDataGenerator()

train_generator = train_datagen.flow_from_directory(
    train_dir,
    target_size = (img_width, img_height),
    batch_size = batch_size,
    class_mode = 'binary',
    shuffle = True)

valid_generator = valid_datagen.flow_from_directory(
    valid_dir,
    target_size = (img_width, img_height),
    batch_size = batch_size,
    class_mode = 'binary',
    shuffle = False)

test_generator = test_datagen.flow_from_directory(
    test_dir,
    target_size = (img_width, img_height),
    batch_size = 1,
    class_mode = None,
    shuffle = False)

mpd = classification_model.fit_generator(
    train_generator,
    steps_per_epoch = no_train_images // batch_size,         # number of images per epoch
    epochs = epochs,                                         # number of iterations over the entire data
    validation_data = valid_generator,
    validation_steps = no_valid_images // batch_size)

Epoch 1/10 373/373 [==============================] - 119s 320ms/step - loss: 0.5214 - acc: 0.7357 - val_loss: 0.2720 - val_acc: 0.8758

Epoch 2/10 373/373 [==============================] - 120s 322ms/step - loss: 0.2485 - acc: 0.8935 - val_loss: 0.0568 - val_acc: 0.9829

Epoch 3/10 373/373 [==============================] - 130s 350ms/step - loss: 0.1427 - acc: 0.9435 - val_loss: 0.0410 - val_acc: 0.9796

Epoch 4/10 373/373 [==============================] - 127s 341ms/step - loss: 0.1053 - acc: 0.9623 - val_loss: 0.0197 - val_acc: 0.9971

Epoch 5/10 373/373 [==============================] - 126s 337ms/step - loss: 0.0817 - acc: 0.9682 - val_loss: 0.0136 - val_acc: 0.9948

Epoch 6/10 373/373 [==============================] - 123s 329ms/step - loss: 0.0665 - acc: 0.9754 - val_loss: 0.0116 - val_acc: 0.9985

Epoch 7/10 373/373 [==============================] - 140s 376ms/step - loss: 0.0518 - acc: 0.9817 - val_loss: 0.0035 - val_acc: 0.9997

Epoch 8/10 373/373 [==============================] - 144s 386ms/step - loss: 0.0539 - acc: 0.9832 - val_loss: 8.9459e-04 - val_acc: 1.0000

Epoch 9/10 373/373 [==============================] - 122s 327ms/step - loss: 0.0434 - acc: 0.9850 - val_loss: 0.0023 - val_acc: 0.9997

Epoch 10/10 373/373 [==============================] - 125s 336ms/step - loss: 0.0513 - acc: 0.9844 - val_loss: 0.0014 - val_acc: 1.0000

valid_generator.batch_size=1
score = classification_model.evaluate_generator(valid_generator, 
                                                no_test_images/batch_size, pickle_safe=False)
test_generator.reset()
scores=classification_model.predict_generator(test_generator, len(test_generator))

print("Loss: ", score[0], "Accuracy: ", score[1])

predicted_class_indices=np.argmax(scores,axis=1)
print(predicted_class_indices)

labels = (train_generator.class_indices)
labelss = dict((v,k) for k,v in labels.items())
predictions = [labelss[k] for k in predicted_class_indices]

filenames=test_generator.filenames
results=pd.DataFrame({"Filename":filenames,
                      "Predictions":predictions})

print(results)

Loss: 5.404246180551993e-06 Accuracy: 1.0

print(predicted_class_indices) - ALL 0

[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

                              Filename Predictions
0      test_folder/video_3_frame10.jpg   without
1    test_folder/video_3_frame1001.jpg   without
2    test_folder/video_3_frame1006.jpg   without
3    test_folder/video_3_frame1008.jpg   without
4    test_folder/video_3_frame1009.jpg   without
5    test_folder/video_3_frame1010.jpg   without
6    test_folder/video_3_frame1013.jpg   without
7    test_folder/video_3_frame1014.jpg   without
8    test_folder/video_3_frame1022.jpg   without
9    test_folder/video_3_frame1023.jpg   without
10    test_folder/video_3_frame103.jpg   without
11   test_folder/video_3_frame1036.jpg   without
12   test_folder/video_3_frame1039.jpg   without
13    test_folder/video_3_frame104.jpg   without
14   test_folder/video_3_frame1042.jpg   without
15   test_folder/video_3_frame1043.jpg   without
16   test_folder/video_3_frame1048.jpg   without
17    test_folder/video_3_frame105.jpg   without
18   test_folder/video_3_frame1051.jpg   without
19   test_folder/video_3_frame1052.jpg   without
20   test_folder/video_3_frame1054.jpg   without
21   test_folder/video_3_frame1055.jpg   without
22   test_folder/video_3_frame1057.jpg   without
23   test_folder/video_3_frame1059.jpg   without
24   test_folder/video_3_frame1060.jpg   without

...just some of the outputs but all 650+ are without class.

This is the output and as you can see all the predicted values are 0 for the without class.

This is my first attempt at using Keras and CNN so any help would be really appreciated.

UPDATE

I have solved this. I am currently working on the accuracy but the main problem is now solved.

This is the line that caused problems.

predicted_class_indices=np.argmax(scores,axis=1)

argmax would be returning the index position of the result but as I was using binary classes and in my final layer I had 1 dense. It will only return a single value so it will always return the first class (0 as the index position). As the network is only set, to return one class.

Changing the following fixed my issue.

  1. Changed the class_mode to 'categorical' for the train and test generators
  2. Changed the final dense layer from 1 to 2 so this will return scores/probabilities for both classes. So when you use argmax, it will return the index position of the top score indicating which class it has predicted.
Improve this question
$\endgroup$
5
  • $\begingroup$ You are overfitting badly(data is scarce I assume, use Augmentations or download similar images and expand) $\endgroup$
    – Aditya
    Feb 19, 2019 at 21:50
  • $\begingroup$ @Aditya I have augmented my images using cv2 in a different script such as flipping images, brightness and this was before I knew you could do the same using the ImageDataGenerator but thanks. Any idea why there is only one class being predicted. I have more than 13000 images for training and 6000 for valid. $\endgroup$
    – vis7
    Feb 19, 2019 at 23:15
  • $\begingroup$ What do your images represent? Are your images separated into folders for train and val splits? $\endgroup$
    – Antonio Jurić
    Feb 20, 2019 at 8:46
  • $\begingroup$ @AntonioJurić The images represent someone with and without a football. The directory structure for both train and valid is the following. Train->without->image1, image2 ->with->image4, image5 Valid->without->image3, image6 ->with->image0, image7 $\endgroup$
    – vis7
    Feb 20, 2019 at 11:06
  • $\begingroup$ Welcome to the site! If you have found a solution, you should consider answering your own question (that is allowed) so that people can benefit from it in the future. $\endgroup$
    – I_Play_With_Data
    Feb 21, 2019 at 18:41

1 Answer 1

Reset to default
4
$\begingroup$

UPDATE

I have solved this. I am currently working on the accuracy but the main problem is now solved.

This is the line that caused problems.

predicted_class_indices=np.argmax(scores,axis=1)

argmax would be returning the index position of the result but as I was using binary classes and in my final layer I had 1 dense. It will only return a single value so it will always return the first class (0 as the index position). As the network is only set, to return one class.

Changing the following fixed my issue.

1.Changed the class_mode to 'categorical' for the train and test generators 2.Changed the final dense layer from 1 to 2 so this will return scores/probabilities for both classes. So when you use argmax, it will return the index position of the top score indicating which class it has predicted.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your answer. You literally saved me hours of head scratching. $\endgroup$
    – mockash
    Aug 14, 2021 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.