Consider a very large data set that doesn't fit into memory. Would I be able to get (nearly) the same behavior from multiple calls to train_test_split when calling train_test_split by passing batches of a source data set as opposed to the whole thing at once?

This code is just hypothetical to illustrate my question.

# X, y is the entire dataset.
x_train, y_train, x_test, y_test = train_test_split(X,y,stratify=y, test_size=.2)

# compared to
for x_bat, y_bat in stream_next_batch_from_file():
    x_train, y_train, x_test, y_test = train_test_split(x_bat, y_bat, stratify=y_bat, test_size=.2)
    # Append the splits to their respective files.
    append_data(x_train, y_train, "train_set_filename")
    append_data(x_test, y_test, "test_set_filename")
    # etc.

1 Answer 1


This should be fine. If the data is independent of its position in the list, this should give basically identical results.

If the data depends on order, then grabbing some data for training and testing from each batch will have a smoothing effect, making the training and test sets more representative than the results you would get with a raw train_test_split.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.