Consider a very large data set that doesn't fit into memory. Would I be able to get (nearly) the same behavior from multiple calls to train_test_split when calling train_test_split by passing batches of a source data set as opposed to the whole thing at once?
This code is just hypothetical to illustrate my question.
# X, y is the entire dataset. x_train, y_train, x_test, y_test = train_test_split(X,y,stratify=y, test_size=.2) # compared to for x_bat, y_bat in stream_next_batch_from_file(): x_train, y_train, x_test, y_test = train_test_split(x_bat, y_bat, stratify=y_bat, test_size=.2) # Append the splits to their respective files. append_data(x_train, y_train, "train_set_filename") append_data(x_test, y_test, "test_set_filename") # etc.