Why is torch DQN updating all state-action values?

Question

I am following the DQN Tutorial from pytorch. If I understand correctly, they update all action values based on the loss and gradients of all actions together there:

    # Compute Huber loss
    criterion = nn.SmoothL1Loss()
    loss = criterion(state_action_values, expected_state_action_values.unsqueeze(1))

    # Optimize the model
    optimizer.zero_grad()
    loss.backward()
    for param in policy_net.parameters():
        param.grad.data.clamp_(-1, 1)
    optimizer.step()

Shouldn't the gradient update only apply to the chosen action? Meaning, if action one has been chosen, shouldn't it only backpropagate the gradients from the output node that corresponds with action 1, without backpropagating from action 0? I understood the DQN setup as such, that each

outputs correspond to the predicted Q-values of the individual action for the input state