For a binary image classification problem (CNN using tf.keras). My image data is separated into folders (train, validation, test) each with subfolders for two balanced classes. Borrowing code from this tutorial, I initially loaded my training and validation sets this way:
train_ds = tf.keras.preprocessing.image_dataset_from_directory( train_path, validation_split=0.2, subset="training", seed=42, image_size=image_size, batch_size=batch_size, ) val_ds = tf.keras.preprocessing.image_dataset_from_directory( train_path, validation_split=0.2, subset="validation", seed=42, image_size=image_size, batch_size=batch_size, )
Note that I am loading both training and validation from the same folder and then using validation_split (because I wanted to play around before using the real validation set). My model was performing quite well, achieving validation accuracy of~0.95.
Then I decided to update my code to load the real validation set:
train_ds = image_dataset_from_directory( train_path, seed=42, image_size=image_size, batch_size=batch_size, ) val_ds = image_dataset_from_directory( val_path, seed=42, image_size=image_size, batch_size=batch_size, )
Now my model is performing substantially worse (~0.75 accuracy). I'm trying to understand why. I suspect my initial code was causing some data leakage. Now that I look at it, I can't tell how the second call of image_dataset_from_directory (for val_ds) knows not to load images that were already loaded for the first call (for train_ds) (unless having the same random seed prevents this). I would be certain this is the issue, except for the fact that I pulled this code directly from a keras.io tutorial - surely they wouldn't make such a basic mistake?
Main question: Given the way that
subset interact with
image_dataset_from_directory(), is the first version of my code resulting in data leakage?
If it should not be resulting in data leakage between training and validation sets, then I will need to consider other possibilities, such as:
- There are actual differences between images in the train and validation set folders. I could combine and reshuffle them.
- The order of images in the training folder is such that given my random seed "easier" images were getting pulled for the validation set.