In the book "hands-on machine learning with scikit-learn and tensorflow: concepts, tools, and techniques to build intelligent systems" , more specifically in chapter 2 , the writer is teaching us how to create a test set. He mention that we need to keep the test set consistent across multiple runs . To do so he mentioned the following :
A common solution is to use each instance’s identifier to decide whether or not it should go in the test set (assuming instances have a unique and immutable identifier). For example, you could compute a hash of each instance’s identifier and put that instance in the test set if the hash is lower or equal to 20% of the maximum hash value. This ensures that the test set will remain consistent across multiple runs, even if you refresh the dataset. The new test set will contain 20% of the new instances, but it will not contain any instance that was previously in the training set. My question is why it is important to do so ? why we don't like to generate different test set for each run ?