3
$\begingroup$

I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, the random seed will reset with each updated dataset and take a different train/test split. Doing this, your ML algos will eventually cover the whole dataset, defeating the purpose of the train/test split because it'll eventually train on too much of the whole dataset over time.

The book I'm reading (Hands-On Machine Learning with Scikit-Learn and Tensorflow) gives this code to split train/test by id:

# Function to check test set's identifier.
def test_set_check(identifier, test_ratio):
    return crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32

# Function to split train/test
def split_train_test_by_id(data, test_ratio, id_column):
    ids = data[id_column]
    in_test_set = ids.apply(lambda id_: test_set_check(id_, test_ratio))
    return data.loc[~in_test_set], data.loc[in_test_set]

And it says when there's no ID column given, to create one either by indexing the rows or creating a unique index from one of the variables.

My questions are:

  1. What is the 3rd line doing:

    crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32

  2. What is the anonymous function doing in the 2nd to last line?

    lambda id_: test_set_check(id_, test_ratio)

  3. In practice, do you commonly split datasets by id in this manner?

Thanks,

Greg

$\endgroup$
  • $\begingroup$ What book are you reading? Does it not explain that anywhere? $\endgroup$ – n1k31t4 May 3 at 23:00
  • $\begingroup$ The book is Hands-On Machine Learning with Scikit-Learn and Tensorflow. It explains the concept briefly, as I tried to summarize in the first paragraph of this post. But problem is the github for the book updates some of the code to account for changes in sklearn .20, but the explanations aren't as good for the changed code. So originally it used import hashlib, now it uses import crc32. And even when it was hashlib, it didn't really explain what hashes are so I don't really get the concept either way. $\endgroup$ – Greg Rosen May 3 at 23:35
  • $\begingroup$ I have the same book A Geron 2017. Could you let me know which section the GitHub code refers too? $\endgroup$ – Michael G. May 4 at 6:39
3
$\begingroup$

In Auriel Geron's book, there is a short description of the approach:

you could compute a hash of each instance’s identifier, keep only the last byte of the hash, and put the instance in the test set if this value is lower or equal to 51 (~20% of 256). This ensures that the test set will remain consistent across multiple runs, even if you refresh the dataset. The new test set will contain 20% of the new instances, but it will not contain any instance that was previously in the training set.

While a full explanation of what exactly happens and why is probably best placed on StackOverflow, I can try to answer your questions, first with some background info.

The method uses a cyclic redundancy check, which is a method of checking that the raw blocks of memory have not been damaged/changed. It is a way to ensure data integrity, e.g. in network traffic - checking if a message way altered between being sent and received.

For train/test splits, it is checking the unique identifier of each sample. We have a column that gives each sample an ID - this should never be changed! Don't delete rows, only append to the end with new unique IDs.

In this part: test_ratio * 2**32, the part $2^{32}$ represents the largest integer of a 32-bit system.

0xFFFFFFFF is a large number; it's the hexadecimal representation of $232^{-1}$

To answer your questions:

  1. What is the 3rd line doing:

crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32

Based on the information I gave above, we see the crc32 function finds the checksum value in memory (of the unique identifier). If we know the unique ID has never changed, then We ensure that crc32(np.int64(identifier)) & 0xffffffff will always return exactly the same numeric value, across all python versions and platforms.

Imagine we give IDs in the range 0-80 for train, and 81-100 for test. No we want to make sure a sample'd s ID falls in the first bucket. We check its ID is simple less that 81, right? Well the numeric value we made above is checked to be less than our test_ratio * 2**32, where 2**32 is the largest 32-bit number. It checks that the sample's ID is within the range of train data, not in the test bucket:: > test_ratio * 2**32.

  1. What is the anonymous function doing in the 2nd to last line?

lambda id_: test_set_check(id_, test_ratio)

This simply applies our test_set_check function to each sample's unique identifiers. Using the apply methd on a Pandas Series object (here it is one column of a Pandas DataFrame).

  1. In practice, do you commonly split datasets by id in this manner?

Not really... Scikit_learn's train_test_split if often good enough. I think there are many other ways to remove bias and errors from your models before worrying too much about the impact of random splits.

For example, the snoop bias, whereby you analyse the entire dataset yourself before deciding on a model architecture/pipeline, thereby incorporating knowledge of the entuire distribution, which is inherently biasing our model.

There is also bias in overfitting e.g. in sequential imaging data (think frames of videos) such that the background is consistent, even though the objects you might want to detect are not. Your model will learn what to expect based on the background, which is not robust! HEre you might look into using a geographical split (i.e. not random at all).


On a side note, there is also a slightly robuster way of setting random seeds in Python (instead of using NumPy's random seed generator). Have a look here for some differences.


Helpful resources:
  1. https://stackoverflow.com/questions/36819849/detect-int32-overflow-using-0xffffffff-masking-in-python
  2. https://pynative.com/python-random-module/
  3. https://stackoverflow.com/questions/30092226/how-to-calculate-crc32-with-python-to-match-online-results
  4. https://stackoverflow.com/questions/49331030/bitwise-xor-0xffffffff/49332291#49332291
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.