19
$\begingroup$

Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ :

validation_split: Float between 0 and 1. Fraction of the training data to be used as validation data. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on this data at the end of each epoch. The validation data is selected from the last samples in the x and y data provided, before shuffling.

Please note the last line:

The validation data is selected from the last samples in the x and y data provided, before shuffling.

Does it means that validation data is always fixed and taken from bottom of main dataset?

Is there any way it can be made to randomly select given fraction of data from main dataset?

$\endgroup$
19
$\begingroup$

You actually would not want to resample your validation set after each epoch. If you did this your model would be trained on every single sample in your dataset and thus this will cause overfitting. You want to always split your data before the training process and then the algorithm should only be trained using the subset of the data for training.

The function as it is designed ensures that the data is separated in such a way that it always trains on the same portion of the data for each epoch. All shuffling is done within the training sample between epochs if that option is chosen.

However, for some datasets getting the last few instances is not useful, specifically if the dataset is regroup based on class. Then the distribution of your classes will be skewed. Thus you will need some kind of random way to extract a subset of the data to get balanced class distributions in the training and validation set. For this I always like to use the sklearn function as follows

from sklearn.model_selection import train_test_split

# Split the data
x_train, x_valid, y_train, y_valid = train_test_split(data, labels, test_size=0.33, shuffle= True)

It's a nice easy to use function that does what you want. The variables data and labels are standard numpy matrices with the first dimension being the instances.

$\endgroup$
  • 2
    $\begingroup$ I did not mean getting different test set after each epoch. I was asking about it getting data from the end of dataset since many dataset may be arranged according to class (as you mentioned). I know about train_test_split and now you confirm that this is a better method since it will randomly get test/validation data from dataset. $\endgroup$ – rnso Sep 30 '18 at 6:54
  • $\begingroup$ @rnso, unfortunately Keras does not provide that option. I guess it's not really within the scope of what they want to offer. Would make the implementation a bit confusing having 2 different method inputs with random in the name. $\endgroup$ – JahKnows Sep 30 '18 at 14:34
  • $\begingroup$ Didn't you mix the definition of validation and test data by using that train_test_spli? $\endgroup$ – bagustris May 21 '19 at 22:05
  • $\begingroup$ If you use the shuffle parameter, would you also use shuffle in the .fit for a keras model? $\endgroup$ – Shamoon May 23 '19 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.