Does validation_split in tf.keras.preprocessing.image_dataset_from_directory result in Data Leakage?

Question

For a binary image classification problem (CNN using tf.keras). My image data is separated into folders (train, validation, test) each with subfolders for two balanced classes. Borrowing code from this tutorial, I initially loaded my training and validation sets this way:

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    train_path,
    validation_split=0.2,
    subset="training",
    seed=42,
    image_size=image_size,
    batch_size=batch_size,
)
val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    train_path,
    validation_split=0.2,
    subset="validation",
    seed=42,
    image_size=image_size,
    batch_size=batch_size,
)

Note that I am loading both training and validation from the same folder and then using validation_split (because I wanted to play around before using the real validation set). My model was performing quite well, achieving validation accuracy of~0.95.

Then I decided to update my code to load the real validation set:

train_ds = image_dataset_from_directory(
    train_path,
    seed=42,
    image_size=image_size,
    batch_size=batch_size,
)

val_ds = image_dataset_from_directory(
    val_path,
    seed=42,
    image_size=image_size,
    batch_size=batch_size,
)

Now my model is performing substantially worse (~0.75 accuracy). I'm trying to understand why. I suspect my initial code was causing some data leakage. Now that I look at it, I can't tell how the second call of image_dataset_from_directory (for val_ds) knows not to load images that were already loaded for the first call (for train_ds) (unless having the same random seed prevents this). I would be certain this is the issue, except for the fact that I pulled this code directly from a keras.io tutorial - surely they wouldn't make such a basic mistake?

Main question: Given the way that validation_split and subset interact with image_dataset_from_directory(), is the first version of my code resulting in data leakage?

If it should not be resulting in data leakage between training and validation sets, then I will need to consider other possibilities, such as:

• There are actual differences between images in the train and validation set folders. I could combine and reshuffle them.
• The order of images in the training folder is such that given my random seed "easier" images were getting pulled for the validation set.

Answer 1

A possible issue is that Keras validation_split uses the "last $x$ percent" of data as validation data without shuffling the data. So if your data has a certain stratification, this stratification will affect the validation set.

I further understand from the docs that the shuffle argument in .fit() does not shuffle data before assigning the validation data. It shuffles training data before each epoch.

As far as I remember I had a similar problem and needed to "manually" shuffle my data before feeding it to the NN in order to avoid problematic bunching of classes in the validation set (defined by validation_split).

From the docs:

validation_split
Float between 0 and 1. Fraction of the training data to be used as validation data. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on this data at the end of each epoch. The validation data is selected from the last samples in the x and y data provided, before shuffling.

shuffle
Logical (whether to shuffle the training data before each epoch) or string (for "batch"). "batch" is a special option for dealing with the limitations of HDF5 data; it shuffles in batch-sized chunks. Has no effect when steps_per_epoch is not NULL.