0
$\begingroup$

so I'm following the official keras tutorial here. However I couldn't really understand the subset and validation_split arguments of tf.keras.preprocessing.text_dataset_from_directory.

How I use them is:

raw_train_ds = tf.keras.preprocessing.text_dataset_from_directory(
    "aclImdb/train",
    batch_size=BATCH_SIZE,
    validation_split=0.2,
    subset="training",
    seed=1337
)

raw_val_ds = tf.keras.preprocessing.text_dataset_from_directory(
    "aclImdb/train",
    batch_size=BATCH_SIZE,
    validation_split=0.2,
    subset="validation",
    seed=1337
)

As I understand we're trying to split train data into validation of its 0.2. However we do the same for both "training" and "validation" subsets on same dataset with the same validation_split value which is 0.2. So isn't it supposed to take 0.2 of training data to raw_train_ds and 0.2 to raw_val_ds instead of 0.8, 0.2?

I checked the documentation from here which was not really useful. I also couldn't find any related questions. How does this work exactly?

$\endgroup$

1 Answer 1

1
$\begingroup$

Not sure if this is still relevant.

As stated in the documentation, 'subset' must be 'training' or 'validation' but is only used if 'validation_split' is used. But the documentation does not mention what the link between the value in 'validation_split' and 'subset' is.

If the subset is 'validation' then the value in 'validation_split' is used directly to split the dataset. Ex: If the dataset is 100 and the split is 0.2 and subset is validation then the size of the validation set will be 20 (0.2 x 100).

On the other hand, if the subset is 'training' then the size of the training data set will be (1-validation_split) x data set. Ex: If the dataset is 100 and the split is 0.2 and subset is training then the size of the training set will be 80 ((1-0.2) x 100).

This is really a bad design :-)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.