1
$\begingroup$

After reading a lot of stuff, I'm still not sure how to calculate the priorities for Prioritized Experience Replay (PER).

Example code taken from here

def compute_td_loss(batch_size, beta):
    state, action, reward, next_state, done, indices, weights = replay_buffer.sample(batch_size, beta) 

    state      = Variable(torch.FloatTensor(np.float32(state)))
    next_state = Variable(torch.FloatTensor(np.float32(next_state)))
    action     = Variable(torch.LongTensor(action))
    reward     = Variable(torch.FloatTensor(reward))
    done       = Variable(torch.FloatTensor(done))
    weights    = Variable(torch.FloatTensor(weights))

    q_values      = current_model(state)
    next_q_values = target_model(next_state)

    q_value          = q_values.gather(1, action.unsqueeze(1)).squeeze(1)
    next_q_value     = next_q_values.max(1)[0]
    expected_q_value = reward + gamma * next_q_value * (1 - done)
    
    loss  = (q_value - expected_q_value.detach()).pow(2) * weights
    prios = loss + 1e-5
    loss  = loss.mean()
        
    optimizer.zero_grad()
    loss.backward()
    replay_buffer.update_priorities(indices, prios.data.cpu().numpy())
    optimizer.step()
    
    return loss

(V1) As you can see, the weights are used to calculate the loss and the prios. The loss is the mean squared error including the weights?

loss  = (q_value - expected_q_value.detach()).pow(2) * weights
prios = loss + 1e-5
loss  = loss.mean()

(V2) In "Hands-On Reinforcement Learning for Games: Implementing self-learning agents" on page 176 they use ONE value for ALL prios, coming from the first pow(2).mean():

loss  = (q_value - expected_q_value.detach()).pow(2).mean()
prios = loss + 1e-5
loss  = loss.mean()

(v3) Here loss is the mean squared error, prios are the raw errors:

terr  = (current_Q - target_Q.detach())
prios = terr + 1e-5
loss  = terr.pow(2).mean()

(v4) Here loss is the mean squared error, prios are the squared errors:

terr  = (current_Q - target_Q.detach())
prios = terr.pow(2) + 1e-5
loss  = terr.pow(2).mean()

(v5) Here loss is the mean squared error, prios are the absolute errors:

terr  = (current_Q - target_Q.detach())
prios = torch.abs(terr) + 1e-5
loss  = terr.pow(2).mean()

(v6) Here loss is the mean squared error, prios are the squared absolute errors:

terr  = (current_Q - target_Q.detach())
prios = torch.abs(terr).pow(2) + 1e-5
loss  = terr.pow(2).mean()

Which version is correct and why?

Thanks in advance!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.