I am following the DQN Tutorial from pytorch. If I understand correctly, they update all action values based on the loss and gradients of all actions together there:

    # Compute Huber loss
    criterion = nn.SmoothL1Loss()
    loss = criterion(state_action_values, expected_state_action_values.unsqueeze(1))

    # Optimize the model
    for param in policy_net.parameters():
        param.grad.data.clamp_(-1, 1)

Shouldn't the gradient update only apply to the chosen action? Meaning, if action one has been chosen, shouldn't it only backpropagate the gradients from the output node that corresponds with action 1, without backpropagating from action 0? I understood the DQN setup as such, that each

outputs correspond to the predicted Q-values of the individual action for the input state



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.