# How do I efficiently load data from disk during training of deep learning models in pytorch?

I'm trying to train a deep learning model without loading the entire dataset into memory. My main question is, what's the best way of doing this?

It seems like HDF5 is a common method that people accomplish this, and is what I tried first. However when using pytorch's dataloader class, this ran extremely slowly. I created my own iterator which ran faster, however the data is not randomized every batch. I'm trying to understand why the pytorch dataloader is running slowly and if there is something I can do about it.

Below is my code

First I defined a dataset class that takes in a filepath to an HDF5 dataset. My understanding of this code is that it reads from disk whenever getitem is called.

class My_H5Dataset(torch.utils.data.Dataset):

def __init__(self, file_path):
super(My_H5Dataset, self).__init__()
h5_file = h5py.File(file_path , 'r')
self.features = h5_file['features']
self.labels = h5_file['labels']
self.index = h5_file['index']
self.labels_values = h5_file['labels_values']
self.index_values = h5_file['index_values']

def __getitem__(self, index):

return (torch.from_numpy(self.features[index,:]).float(),
torch.from_numpy(self.labels[index,:]).float(),
torch.from_numpy(self.index[index,:]).float(),
torch.from_numpy(np.array(self.labels_values[index])),
torch.from_numpy(np.array(self.index_values[index])))

def __len__(self):
return self.features.shape[0]


Then I simply pass this into a pytorch dataloader as follows

train_dataset = My_H5Dataset(hdf5_data_folder_train)
train_ms = MySampler(train_dataset)
sampler=train_ms,num_workers=2)


My other method was to manually define an iterator. And this does run much faster. However I have not noticed a speed difference between the data stored in an HDD versus an SSD which makes me worried that there is a bottleneck somewhere that I am missing.

def hdf5_loader_generator(dataset, batch_size, as_tensor=True, n_samples = 10000):
"""Given an h5 path to a file that holds the arrays, returns a generator
that can get certain data at a time."""
stop = n_samples
curr_index = start = 0
while 1:
stop_index = min([curr_index + batch_size, stop])
ft, sp, index, sp_values, index_values = dataset[curr_index:stop_index]
curr_index += batch_size
if curr_index >= stop:
curr_index = start
continue

yield ft, sp, index, sp_values, index_values

def hdf5_data_iterator(dataset, batch_size, as_tensor=True):



I timed pytorch's iterator as follows:


start = time.time()
for i, data in enumerate(trainloader, 0):
ft, sp, index, sp_values, index_values = data
ft, sp, index = ft.to(device), sp.to(device), index.to(device)
if i > 500:
break

end = time.time()

print(end-start)


I timed my iterator as follows:


train_dataset = My_H5Dataset(hdf5_data_folder_train)
samples = hdf5_data_iterator(train_dataset, batch_sizes)
start = time.time()
for i, data in enumerate(samples):
ft, sp, index, sp_values, index_values = data
ft, sp, index = ft.to(device), sp.to(device), index.to(device)
if i > 500:
break

end = time.time()

print(end-start)


Is there a better approach than what I am currently trying?

Thank you for the help!

• Hi @Pedro , I find your question very interesting, I'm looking for the same Feb 16 at 6:18