6
$\begingroup$

I've set the seeds like this (hoping to cover all bases):

random.seed(666)
np.random.seed(666)
torch.manual_seed(666)
torch.cuda.manual_seed_all(666)
torch.backends.cudnn.deterministic = True

The below code will still output DIFFERENT batches for both namesTrainLoader1 and namesTrainLoader2 but they should really be the same. How come creating the model is affecting the deterministic values?

namesDataset = NamesDataset()
namesTrainLoader1 = DataLoader(namesDataset, batch_size=5, shuffle=True)
for each in namesTrainLoader1:
    print(each)

model = TorchRNN(inputSize, hiddenSize, outputSize)

namesTrainLoader2 = DataLoader(namesDataset, batch_size=5, shuffle=True)
for each in namesTrainLoader2:
    print(each)

Output for namesTrainLoader1:

('saiki', 'close', 'sloan', 'horos', 'roman')
...

Output for namesTrainLoader2:

('david', 'abeln', 'hatit', 'holan', 'protz')
...

I also tried using worker_init_fn (e.g. with lambda x: 0) in the DataLoader, but that made no difference.

Why is this not deterministic? How can I make it deterministic? i.e. reset the internal seed of the DataLoader?

$\endgroup$
  • $\begingroup$ try shuffle as false? $\endgroup$ – Aditya Jun 4 '19 at 10:39
  • 1
    $\begingroup$ That doesn't really solve the problem, since I do want the data to be shuffled. I just want it to be deterministic. $\endgroup$ – Muppet Jun 4 '19 at 17:01
  • $\begingroup$ Have you tried setting the see before launching the python? $\endgroup$ – Aditya Jun 4 '19 at 17:10
  • 1
    $\begingroup$ Before launching Python? What do you mean? The seed is set at the beginning of the script, as noted in the question $\endgroup$ – Muppet Jun 4 '19 at 17:29
  • $\begingroup$ What about trying to write a custom sampler?(thanks to kostia) Plus can you open an issue on their Repo with minimal code? And drop a link here because it might be that they are re-seeding the seed internally again.. $\endgroup$ – Aditya Jun 5 '19 at 3:17
2
$\begingroup$

If you want to shuffle the data in a deterministic way, how about shuffling the dataset beforehand e.g. in a simple list of filenames, then simply reading that list deterministically in a single-processed loop, with shuffle = False in the DataLoader??

Another things that may cause non-deterministic behaviour is using multiple processes - then there are operations that are passed out and managed by the operating system, which doesn't pay attention to any of the random seeds you set. Performance is dependent on available resources i.e. affected by other activities running on your host machine.

In addition to that, any interaction between CPU and GPU could be causing non-deterministic behaviour, as data transfer is non-deterministic (related Nvidia thread). Data packets can be split differently every time, but there are apparent CUDA-level solutions in the pipeline.

$\endgroup$
0
$\begingroup$

I came into the same problem while using a DataLoader.

In my opinion, this could be derived from the initialization of the model. Since the model parameters need random numbers to init, the random number generator could be changed because the model took some numbers from it. Therefore, shuffling the dataset after the initialization of a model may lead to a different order.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.