46
$\begingroup$

Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ :

validation_split: Float between 0 and 1. Fraction of the training data to be used as validation data. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on this data at the end of each epoch. The validation data is selected from the last samples in the x and y data provided, before shuffling.

Please note the last line:

The validation data is selected from the last samples in the x and y data provided, before shuffling.

Does it means that validation data is always fixed and taken from bottom of main dataset?

Is there any way it can be made to randomly select given fraction of data from main dataset?

$\endgroup$

2 Answers 2

37
$\begingroup$

You actually would not want to resample your validation set after each epoch. If you did this your model would be trained on every single sample in your dataset and thus this will cause overfitting. You want to always split your data before the training process and then the algorithm should only be trained using the subset of the data for training.

The function as it is designed ensures that the data is separated in such a way that it always trains on the same portion of the data for each epoch. All shuffling is done within the training sample between epochs if that option is chosen.

However, for some datasets getting the last few instances is not useful, specifically if the dataset is regroup based on class. Then the distribution of your classes will be skewed. Thus you will need some kind of random way to extract a subset of the data to get balanced class distributions in the training and validation set. For this I always like to use the sklearn function as follows

from sklearn.model_selection import train_test_split

# Split the data
x_train, x_valid, y_train, y_valid = train_test_split(data, labels, test_size=0.33, shuffle= True)

It's a nice easy to use function that does what you want. The variables data and labels are standard numpy matrices with the first dimension being the instances.

$\endgroup$
4
  • 4
    $\begingroup$ I did not mean getting different test set after each epoch. I was asking about it getting data from the end of dataset since many dataset may be arranged according to class (as you mentioned). I know about train_test_split and now you confirm that this is a better method since it will randomly get test/validation data from dataset. $\endgroup$
    – rnso
    Sep 30, 2018 at 6:54
  • $\begingroup$ @rnso, unfortunately Keras does not provide that option. I guess it's not really within the scope of what they want to offer. Would make the implementation a bit confusing having 2 different method inputs with random in the name. $\endgroup$
    – JahKnows
    Sep 30, 2018 at 14:34
  • $\begingroup$ Didn't you mix the definition of validation and test data by using that train_test_spli? $\endgroup$
    – sugab
    May 21, 2019 at 22:05
  • 1
    $\begingroup$ If you use the shuffle parameter, would you also use shuffle in the .fit for a keras model? $\endgroup$
    – Shamoon
    May 23, 2019 at 16:34
3
$\begingroup$

Following the answer from JahKnows, I should point out that if you want a fixed validation dataset which is chosen after shuffling, you can use the train_test_split method to get your separate validation dataset and then use the validation_data argument in the fit method instead of validation_split, and point to the x and y of your validation data.

If you want both validation and test datasets, you can use the train_test_split method twice, like this:

from sklearn.model_selection import train_test_split

# Separate the test data
x, x_test, y, y_test = train_test_split(x, y, test_size=0.15, shuffle=True)

# Split the remaining data to train and validation
x_train, x_val, y_train, y_val = train_test_split(x, y, test_size=0.15, shuffle=True)

...

# Training the Keras model
the_model.fit(x=x_train, y=y_train, batch_size=64, epochs=100, validation_data=(x_val, y_val))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.