CNN model with images - 100% accuracy on validation and test sets with limited data?

Question

I have a binary classification problem (e.g. if there is a human in the room or not) with a small dataset of images from a thermal camera. Originally, those were 7 videos, which I have converted into images of the 224 x 224 pixels. The images are of a still man sitting on a chair, two men sitting on a chair(there are very clear thermal outlines), or an empty chair(mostly a dark background).

I have about 205 images. so I have 76 no human images with label 0 (the empty chair) and 129 human images with label 1 (one man or two men).

I have used a train_test_split() to split these into the test and train datasets proportionally using stratify for an equal representation of labels 0 and 1, and then into the train and validation datasets :

from sklearn.model_selection import train_test_split    
X_train_images, X_test_images, y_train_labels, y_test_labels = train_test_split(all_images, all_labels_binary, stratify= all_labels_binary, random_state=0, test_size = .10)       
X_train_images, X_val_images, y_train_labels, y_val_labels = train_test_split(X_train_images, y_train_labels,stratify= y_train_labels, test_size=0.30)

My shapes are:

Train: X_train_images=(96, 224, 224, 3), y_train_labels=(96, 1)
Validation: X_val_images=(42, 224, 224, 3), y_val_labels=(42, 1)
Test: X_test_images=(21, 224, 224, 3), y_test_labels=(21, 1)

then I did a similar split to separate into a test and training set.

My model is defined as follows:

cnn_model = tf.keras.Sequential([
    Conv2D(32, (5,5), padding = 'same', activation = 'relu', input_shape =(input_shape)),
    Conv2D(64, (3,3), padding = 'same', activation = 'relu', input_shape =(input_shape)),
    MaxPooling2D(2,2),
    Flatten(),
    Dense(64, activation = 'relu'),
    Dense(1, activation = "sigmoid"),                       

])

cnn_model.summary()
    
   Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 224, 224, 32)      2432      
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 224, 224, 64)      18496     
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 112, 112, 64)      0         
_________________________________________________________________
flatten (Flatten)            (None, 802816)            0         
_________________________________________________________________
dense (Dense)                (None, 64)                51380288  
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 65        
=================================================================
Total params: 51,401,281
Trainable params: 51,401,281
Non-trainable params: 0

Now I am having a perfect fit with my validation and test datasets according to my accuracy and loss graphs.

Can someone explain to me why there is a perfect fit (100% accuracy), is it because I have very limited data? what can I do to make my results more meaningful? if I split my videos into more images (say every 3 seconds instead of every 10 seconds), will my results still be the same?

I wonder if try some more models like VGG16 or Resnet in keras, will my results will still be the same with no difference between the models?

Thank you for any advice.

Answer 1

If you see the model summary, it has been found that there is no non-trainable data. This suggests that you are training for the whole dataset. And, probably that's why you are getting 100% accuracy.

Answer 2

Taking 207 images made from 7 videos, and then randomly forming train and validation sets can lead to a massive data leakage: both train and validation sets will have many images from every video.

For example, if one video is a person mostly sitting, the other one is mostly empty chair, etc., then instead of learning to solve the actual problem the model will just easily learn which video corresponds to which label, for example, by using some simple colour information.

To do it properly you need to make sure the images in your training, validation and test sets are from different videos.