When I include validation_data=(x_val, y_val) in model.fit_generator, should I create another test dataset for accuracy measures?

Question

While modelling in keras, often I see the usage of validation_data=(x_val, y_val) in model.fit_generator where (x_val, y_val) generally forms 10% of the dataset. While training, is it that the model takes hint from the validation loss calculated on (x_val, y_val) and I need to create another test_data for measuring accuracies in the end? Or, I can use the same (x_val, y_val) in input of model.fit_generator and measuring the accuracy in the end.

The confusion arises from the fact that we are often advised to create training, validation, test datasets while modelling. If the validation dataset has been used to judge some parameters like when to stop(early stopping), etc; wouldn't it be unfair to use the validation_data to calculate the various measures of accuracy?

Answer 1

You are correct in saying that it would be unfair - and if avoidable, you shouldn't do it.

In order to truly be able to claim (in a statistical sense) that a model achieves e.g. 90% accuracy, the test must be performed on unseen data. That is where your test data should be used.

Training a neural network requires the validation data (as you mentioned, within the fit_generator method of a Keras model), in order to compute errors and steer the weights in the right direction. The final accuracy you report needs to be on data to which the the training pipeline has never been exposed.

Creating a training/validation/test split is advised where possible; however, it can sometimes be a challenge, due to things like lack of data and imbalanced datasets. You can try things such as cross-validation - here is an example using Keras. Her is another example, in a similar question.