I'm trying to train a deep learning model without loading the entire dataset into memory. My main question is, what's the best way of doing this?

It seems like HDF5 is a common method that people accomplish this, and is what I tried first. However when using pytorch's dataloader class, this ran extremely slowly. I created my own iterator which ran faster, however the data is not randomized every batch. I'm trying to understand why the pytorch dataloader is running slowly and if there is something I can do about it.

Below is my code

First I defined a dataset class that takes in a filepath to an HDF5 dataset. My understanding of this code is that it reads from disk whenever getitem is called.

class My_H5Dataset(torch.utils.data.Dataset):

def __init__(self, file_path):
    super(My_H5Dataset, self).__init__()
    h5_file = h5py.File(file_path , 'r')
    self.features = h5_file['features']
    self.labels = h5_file['labels']
    self.index = h5_file['index']
    self.labels_values = h5_file['labels_values']
    self.index_values = h5_file['index_values']

def __getitem__(self, index): 

    return (torch.from_numpy(self.features[index,:]).float(),

def __len__(self):
    return self.features.shape[0]

Then I simply pass this into a pytorch dataloader as follows

train_dataset = My_H5Dataset(hdf5_data_folder_train)
train_ms = MySampler(train_dataset)
trainloader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, 

My other method was to manually define an iterator. And this does run much faster. However I have not noticed a speed difference between the data stored in an HDD versus an SSD which makes me worried that there is a bottleneck somewhere that I am missing.

def hdf5_loader_generator(dataset, batch_size, as_tensor=True, n_samples = 10000):
    """Given an h5 path to a file that holds the arrays, returns a generator
    that can get certain data at a time."""
    stop = n_samples
    curr_index = start = 0
    while 1:
        stop_index = min([curr_index + batch_size, stop])
        ft, sp, index, sp_values, index_values = dataset[curr_index:stop_index]
        curr_index += batch_size
        if curr_index >= stop:
            curr_index = start

        yield ft, sp, index, sp_values, index_values
def hdf5_data_iterator(dataset, batch_size, as_tensor=True):
    return iter(hdf5_loader_generator(dataset, batch_size, as_tensor))

I timed pytorch's iterator as follows:

start = time.time()
for i, data in enumerate(trainloader, 0):
    ft, sp, index, sp_values, index_values = data
    ft, sp, index = ft.to(device), sp.to(device), index.to(device)
    if i > 500:
end = time.time()


I timed my iterator as follows:

train_dataset = My_H5Dataset(hdf5_data_folder_train)
samples = hdf5_data_iterator(train_dataset, batch_sizes)
start = time.time()
for i, data in enumerate(samples):
    ft, sp, index, sp_values, index_values = data
    ft, sp, index = ft.to(device), sp.to(device), index.to(device)
    if i > 500:
end = time.time()


Is there a better approach than what I am currently trying?

Thank you for the help!

  • $\begingroup$ Hi @Pedro , I find your question very interesting, I'm looking for the same $\endgroup$ Feb 16 at 6:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.