How does the validation_split parameter of Keras' fit function work?

Question

Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ :

validation_split: Float between 0 and 1. Fraction of the training data to be used as validation data. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on this data at the end of each epoch. The validation data is selected from the last samples in the x and y data provided, before shuffling.

Please note the last line:

The validation data is selected from the last samples in the x and y data provided, before shuffling.

Does it means that validation data is always fixed and taken from bottom of main dataset?

Is there any way it can be made to randomly select given fraction of data from main dataset?

Answer 1

You actually would not want to resample your validation set after each epoch. If you did this your model would be trained on every single sample in your dataset and thus this will cause overfitting. You want to always split your data before the training process and then the algorithm should only be trained using the subset of the data for training.

The function as it is designed ensures that the data is separated in such a way that it always trains on the same portion of the data for each epoch. All shuffling is done within the training sample between epochs if that option is chosen.

However, for some datasets getting the last few instances is not useful, specifically if the dataset is regroup based on class. Then the distribution of your classes will be skewed. Thus you will need some kind of random way to extract a subset of the data to get balanced class distributions in the training and validation set. For this I always like to use the sklearn function as follows

from sklearn.model_selection import train_test_split

# Split the data
x_train, x_valid, y_train, y_valid = train_test_split(data, labels, test_size=0.33, shuffle= True)

It's a nice easy to use function that does what you want. The variables data and labels are standard numpy matrices with the first dimension being the instances.

Answer 2

Following the answer from JahKnows, I should point out that if you want a fixed validation dataset which is chosen after shuffling, you can use the train_test_split method to get your separate validation dataset and then use the validation_data argument in the fit method instead of validation_split, and point to the x and y of your validation data.

If you want both validation and test datasets, you can use the train_test_split method twice, like this:

from sklearn.model_selection import train_test_split

# Separate the test data
x, x_test, y, y_test = train_test_split(x, y, test_size=0.15, shuffle=True)

# Split the remaining data to train and validation
x_train, x_val, y_train, y_val = train_test_split(x, y, test_size=0.15, shuffle=True)

...

# Training the Keras model
the_model.fit(x=x_train, y=y_train, batch_size=64, epochs=100, validation_data=(x_val, y_val))