1
$\begingroup$

I am training a LSTM network on CPU and can achieve deterministic results when not using a dataloader. But when I used Pytorchs dataloader I achieve non-deterministic training error results, despite the actual batches being loaded from the dataloader being deterministic.

I have set pretty much every seed I could think of here

random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False  
torch.backends.cudnn.enabled   = False

The two pieces of code are:

optimiser = torch.optim.Adam(model_test.parameters(), learning_rate)
set_seed(42)

for t in range(num_epochs):

    for batch_idx, (X_train, y_train) in enumerate(train_loader):

        # Zero out gradients
        optimiser.zero_grad()

        # Forward pass
        y_pred = model_test(X_train)

        # Loss Function
        loss = loss_fn(y_pred, y_train)

        # Backward pass
        loss.backward()

        # Update parameters
        optimiser.step()

    if t % 100 == 0: print("Epoch ", t, "MSE: ", loss.item())

and

optimiser = torch.optim.Adam(model_test.parameters(), learning_rate)
set_seed(42)

for t in range(num_epochs):

    # Zero out gradients
    optimiser.zero_grad()

    # Forward pass
    y_pred = model_test(X_train)

    # Loss Function
    loss = loss_fn(y_pred, y_train)

    # Backward pass
    loss.backward()

    # Update parameters
    optimiser.step()

    if t % 100 == 0: print("Epoch ", t, "MSE: ", loss.item())

I have seen some posts on Github talking about there being issues with determinism on GPU, but this is just on CPU.

$\endgroup$
1
$\begingroup$

The cudnn implementaton of the LSTM has determinism issues that appear to be fixed in the 7.6.1 release. Check your cudnn version.

https://github.com/pytorch/pytorch/issues/18110

$\endgroup$
0
$\begingroup$

It can be made deterministic by adding set_seed(42) after optimiser.zero_grad(). Not sure what happens in optimiser.zero_grad() to mess with the seeding on every batch.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.