1
$\begingroup$

For a binary image classification problem (CNN using tf.keras). My image data is separated into folders (train, validation, test) each with subfolders for two balanced classes. Borrowing code from this tutorial, I initially loaded my training and validation sets this way:

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    train_path,
    validation_split=0.2,
    subset="training",
    seed=42,
    image_size=image_size,
    batch_size=batch_size,
)
val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    train_path,
    validation_split=0.2,
    subset="validation",
    seed=42,
    image_size=image_size,
    batch_size=batch_size,
)

Note that I am loading both training and validation from the same folder and then using validation_split (because I wanted to play around before using the real validation set). My model was performing quite well, achieving validation accuracy of~0.95.

Then I decided to update my code to load the real validation set:

train_ds = image_dataset_from_directory(
    train_path,
    seed=42,
    image_size=image_size,
    batch_size=batch_size,
)

val_ds = image_dataset_from_directory(
    val_path,
    seed=42,
    image_size=image_size,
    batch_size=batch_size,
)

Now my model is performing substantially worse (~0.75 accuracy). I'm trying to understand why. I suspect my initial code was causing some data leakage. Now that I look at it, I can't tell how the second call of image_dataset_from_directory (for val_ds) knows not to load images that were already loaded for the first call (for train_ds) (unless having the same random seed prevents this). I would be certain this is the issue, except for the fact that I pulled this code directly from a keras.io tutorial - surely they wouldn't make such a basic mistake?

Main question: Given the way that validation_split and subset interact with image_dataset_from_directory(), is the first version of my code resulting in data leakage?

If it should not be resulting in data leakage between training and validation sets, then I will need to consider other possibilities, such as:

  • There are actual differences between images in the train and validation set folders. I could combine and reshuffle them.
  • The order of images in the training folder is such that given my random seed "easier" images were getting pulled for the validation set.
$\endgroup$
2
  • $\begingroup$ validation split in Keras always uses the last $x$ percent of data as validation set. Could this be part of the problem? $\endgroup$
    – Peter
    Nov 27 '21 at 23:49
  • $\begingroup$ @Peter - I had thought the shuffle=True default meant that the images are shuffled before the validation_split, but if the shuffle happens afterwords then this would definitely explain the issue (filenames of images are their lat/long coordinates so the last x images would all be from the same area). Do you know what order these operations happen in? $\endgroup$ Nov 28 '21 at 18:20
0
$\begingroup$

A possible issue is that Keras validation_split uses the "last $x$ percent" of data as validation data without shuffling the data. So if your data has a certain stratification, this stratification will affect the validation set.

I further understand from the docs that the shuffle argument in .fit() does not shuffle data before assigning the validation data. It shuffles training data before each epoch.

As far as I remember I had a similar problem and needed to "manually" shuffle my data before feeding it to the NN in order to avoid problematic bunching of classes in the validation set (defined by validation_split).

From the docs:

validation_split
Float between 0 and 1. Fraction of the training data to be used as validation data. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on this data at the end of each epoch. The validation data is selected from the last samples in the x and y data provided, before shuffling.

shuffle
Logical (whether to shuffle the training data before each epoch) or string (for "batch"). "batch" is a special option for dealing with the limitations of HDF5 data; it shuffles in batch-sized chunks. Has no effect when steps_per_epoch is not NULL.

$\endgroup$
1
  • $\begingroup$ Yes - this is the issue. The filenames of my training data begin with latitude/longitude coordinates so validation_split was selecting a subset of images that are bunched together (ie not random). Thanks for the explanation! $\endgroup$ Nov 28 '21 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.