In the book Hands-on machine learning with scikit-learn and tensorflow: concepts, tools, and techniques to build intelligent systems, more specifically in Chapter 2, the writer is teaching us how to create a test set. He mentions that we need to keep the test set consistent across multiple runs. To do so, he mentioned the following:
A common solution is to use each instance’s identifier to decide whether or not it should go in the test set (assuming instances have a unique and immutable identifier). For example, you could compute a hash of each instance’s identifier and put that instance in the test set if the hash is lower or equal to 20% of the maximum hash value. This ensures that the test set will remain consistent across multiple runs, even if you refresh the dataset. The new test set will contain 20% of the new instances, but it will not contain any instance that was previously in the training set.
My question is why is it important to do so? Why do we not like to generate different test sets for each run?