0
$\begingroup$

I have a binary classification problem (e.g. if there is a human in the room or not) with a small dataset of images from a thermal camera. Originally, those were 7 videos, which I have converted into images of the 224 x 224 pixels. The images are of a still man sitting on a chair, two men sitting on a chair(there are very clear thermal outlines), or an empty chair(mostly a dark background).

I have about 205 images. so I have 76 no human images with label 0 (the empty chair) and 129 human images with label 1 (one man or two men).

I have used a train_test_split() to split these into the test and train datasets proportionally using stratify for an equal representation of labels 0 and 1, and then into the train and validation datasets :

from sklearn.model_selection import train_test_split    
X_train_images, X_test_images, y_train_labels, y_test_labels = train_test_split(all_images, all_labels_binary, stratify= all_labels_binary, random_state=0, test_size = .10)       
X_train_images, X_val_images, y_train_labels, y_val_labels = train_test_split(X_train_images, y_train_labels,stratify= y_train_labels, test_size=0.30)

My shapes are:

Train: X_train_images=(96, 224, 224, 3), y_train_labels=(96, 1)
Validation: X_val_images=(42, 224, 224, 3), y_val_labels=(42, 1)
Test: X_test_images=(21, 224, 224, 3), y_test_labels=(21, 1)

then I did a similar split to separate into a test and training set.

My model is defined as follows:

cnn_model = tf.keras.Sequential([
    Conv2D(32, (5,5), padding = 'same', activation = 'relu', input_shape =(input_shape)),
    Conv2D(64, (3,3), padding = 'same', activation = 'relu', input_shape =(input_shape)),
    MaxPooling2D(2,2),
    Flatten(),
    Dense(64, activation = 'relu'),
    Dense(1, activation = "sigmoid"),                       

])

cnn_model.summary()
    
   Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 224, 224, 32)      2432      
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 224, 224, 64)      18496     
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 112, 112, 64)      0         
_________________________________________________________________
flatten (Flatten)            (None, 802816)            0         
_________________________________________________________________
dense (Dense)                (None, 64)                51380288  
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 65        
=================================================================
Total params: 51,401,281
Trainable params: 51,401,281
Non-trainable params: 0

Now I am having a perfect fit with my validation and test datasets according to my accuracy and loss graphs.

enter image description here

Can someone explain to me why there is a perfect fit (100% accuracy), is it because I have very limited data? what can I do to make my results more meaningful? if I split my videos into more images (say every 3 seconds instead of every 10 seconds), will my results still be the same?

I wonder if try some more models like VGG16 or Resnet in keras, will my results will still be the same with no difference between the models?

Thank you for any advice.

$\endgroup$
6
  • 1
    $\begingroup$ Did you check the image content? It seems as if all the images are the same... $\endgroup$ Jun 24, 2022 at 16:18
  • $\begingroup$ P.S. the ipynb files there relate to a different dataset. thank you for replying as I still cant figure this out! $\endgroup$
    – Bluetail
    Jun 24, 2022 at 16:36
  • $\begingroup$ I have now increased the sample size to 205.. still the same result with accuracy and loss though. $\endgroup$
    – Bluetail
    Jun 24, 2022 at 16:44
  • 3
    $\begingroup$ The example images suggest that the task is intuitively very easy - there's a person if any pixels read hotter than 32 degrees, otherwise there isn't. 100% accuracy is often unusual, but may not be outlandish for simple problems. $\endgroup$ Jun 24, 2022 at 16:55
  • 1
    $\begingroup$ Based on the images this indeed seems like a very easy task for a neural network to solve, especially given the number of parameters that the model has. $\endgroup$
    – Oxbowerce
    Jun 24, 2022 at 17:13

2 Answers 2

0
$\begingroup$

If you see the model summary, it has been found that there is no non-trainable data. This suggests that you are training for the whole dataset. And, probably that's why you are getting 100% accuracy.

$\endgroup$
0
$\begingroup$

Taking 207 images made from 7 videos, and then randomly forming train and validation sets can lead to a massive data leakage: both train and validation sets will have many images from every video.

For example, if one video is a person mostly sitting, the other one is mostly empty chair, etc., then instead of learning to solve the actual problem the model will just easily learn which video corresponds to which label, for example, by using some simple colour information.

To do it properly you need to make sure the images in your training, validation and test sets are from different videos.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.