I'm trying to train a deep learning model without loading the entire dataset into memory. My main question is, what's the best way of doing this?
It seems like HDF5 is a common method that people accomplish this, and is what I tried first. However when using pytorch's dataloader class, this ran extremely slowly. I created my own iterator which ran faster, however the data is not randomized every batch. I'm trying to understand why the pytorch dataloader is running slowly and if there is something I can do about it.
Below is my code
First I defined a dataset class that takes in a filepath to an HDF5 dataset. My understanding of this code is that it reads from disk whenever getitem is called.
class My_H5Dataset(torch.utils.data.Dataset): def __init__(self, file_path): super(My_H5Dataset, self).__init__() h5_file = h5py.File(file_path , 'r') self.features = h5_file['features'] self.labels = h5_file['labels'] self.index = h5_file['index'] self.labels_values = h5_file['labels_values'] self.index_values = h5_file['index_values'] def __getitem__(self, index): return (torch.from_numpy(self.features[index,:]).float(), torch.from_numpy(self.labels[index,:]).float(), torch.from_numpy(self.index[index,:]).float(), torch.from_numpy(np.array(self.labels_values[index])), torch.from_numpy(np.array(self.index_values[index]))) def __len__(self): return self.features.shape
Then I simply pass this into a pytorch dataloader as follows
train_dataset = My_H5Dataset(hdf5_data_folder_train) train_ms = MySampler(train_dataset) trainloader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, sampler=train_ms,num_workers=2)
My other method was to manually define an iterator. And this does run much faster. However I have not noticed a speed difference between the data stored in an HDD versus an SSD which makes me worried that there is a bottleneck somewhere that I am missing.
def hdf5_loader_generator(dataset, batch_size, as_tensor=True, n_samples = 10000): """Given an h5 path to a file that holds the arrays, returns a generator that can get certain data at a time.""" stop = n_samples curr_index = start = 0 while 1: stop_index = min([curr_index + batch_size, stop]) ft, sp, index, sp_values, index_values = dataset[curr_index:stop_index] curr_index += batch_size if curr_index >= stop: curr_index = start continue yield ft, sp, index, sp_values, index_values def hdf5_data_iterator(dataset, batch_size, as_tensor=True): return iter(hdf5_loader_generator(dataset, batch_size, as_tensor))
I timed pytorch's iterator as follows:
start = time.time() for i, data in enumerate(trainloader, 0): ft, sp, index, sp_values, index_values = data ft, sp, index = ft.to(device), sp.to(device), index.to(device) if i > 500: break end = time.time() print(end-start)
I timed my iterator as follows:
train_dataset = My_H5Dataset(hdf5_data_folder_train) samples = hdf5_data_iterator(train_dataset, batch_sizes) start = time.time() for i, data in enumerate(samples): ft, sp, index, sp_values, index_values = data ft, sp, index = ft.to(device), sp.to(device), index.to(device) if i > 500: break end = time.time() print(end-start)
Is there a better approach than what I am currently trying?
Thank you for the help!