Why is PyTorch's DataLoader not deterministic?

Question

I've set the seeds like this (hoping to cover all bases):

random.seed(666)
np.random.seed(666)
torch.manual_seed(666)
torch.cuda.manual_seed_all(666)
torch.backends.cudnn.deterministic = True

The below code will still output DIFFERENT batches for both namesTrainLoader1 and namesTrainLoader2 but they should really be the same. How come creating the model is affecting the deterministic values?

namesDataset = NamesDataset()
namesTrainLoader1 = DataLoader(namesDataset, batch_size=5, shuffle=True)
for each in namesTrainLoader1:
    print(each)

model = TorchRNN(inputSize, hiddenSize, outputSize)

namesTrainLoader2 = DataLoader(namesDataset, batch_size=5, shuffle=True)
for each in namesTrainLoader2:
    print(each)

Output for namesTrainLoader1:

('saiki', 'close', 'sloan', 'horos', 'roman')
...

Output for namesTrainLoader2:

('david', 'abeln', 'hatit', 'holan', 'protz')
...

I also tried using worker_init_fn (e.g. with lambda x: 0) in the DataLoader, but that made no difference.

Why is this not deterministic? How can I make it deterministic? i.e. reset the internal seed of the DataLoader?

Answer 1

If you want to shuffle the data in a deterministic way, how about shuffling the dataset beforehand e.g. in a simple list of filenames, then simply reading that list deterministically in a single-processed loop, with shuffle = False in the DataLoader??

Another things that may cause non-deterministic behaviour is using multiple processes - then there are operations that are passed out and managed by the operating system, which doesn't pay attention to any of the random seeds you set. Performance is dependent on available resources i.e. affected by other activities running on your host machine.

In addition to that, any interaction between CPU and GPU could be causing non-deterministic behaviour, as data transfer is non-deterministic (related Nvidia thread). Data packets can be split differently every time, but there are apparent CUDA-level solutions in the pipeline.

Answer 2

I came into the same problem while using a DataLoader.

In my opinion, this could be derived from the initialization of the model. Since the model parameters need random numbers to init, the random number generator could be changed because the model took some numbers from it. Therefore, shuffling the dataset after the initialization of a model may lead to a different order.