My Directory


lst = [C:\Dataset\image1.png, C:\Dataset\image2.png, C:\Dataset\image3.png ...] 

Can I pass this list to train_test_split instead of loading and input the dataset to the method. If yes, is this a preferable method or should I load the dataset first.

  • $\begingroup$ Did you try passing a list, and what result did you get? $\endgroup$ – Itamar Mushkin Jan 3 at 8:36

You can pass in a list of file paths into scikit-learn's train_test_split.

Here is an example:

from pathlib import Path

from sklearn.model_selection import train_test_split

file_locations = [Path("images1.png"), Path("images2.png")]
X_train, X_test = train_test_split(file_locations)

The choice between loading data all-at-once vs batch-by-batch is context dependent. If you have small amount of data relative to RAM, then loading in the data into memory is preferable because it will be faster and easier to handle. If you do not have enough RAM available, then taking the pointer-to-the-data approach will enable training to happen. The training in this scenario might be slower and require custom code to load the data at the moment it is needed.

  • $\begingroup$ Thank you for the answer @Brian. So, If my ram is not big enough to load the dataset at once, I should use the path of the dataset into train_test split. $\endgroup$ – Shiv Jan 5 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.