# Prioritized Experience Replay - which version is correct?

After reading a lot of stuff, I'm still not sure how to calculate the priorities for Prioritized Experience Replay (PER).

Example code taken from here

def compute_td_loss(batch_size, beta):
state, action, reward, next_state, done, indices, weights = replay_buffer.sample(batch_size, beta)

state      = Variable(torch.FloatTensor(np.float32(state)))
next_state = Variable(torch.FloatTensor(np.float32(next_state)))
action     = Variable(torch.LongTensor(action))
reward     = Variable(torch.FloatTensor(reward))
done       = Variable(torch.FloatTensor(done))
weights    = Variable(torch.FloatTensor(weights))

q_values      = current_model(state)
next_q_values = target_model(next_state)

q_value          = q_values.gather(1, action.unsqueeze(1)).squeeze(1)
next_q_value     = next_q_values.max(1)[0]
expected_q_value = reward + gamma * next_q_value * (1 - done)

loss  = (q_value - expected_q_value.detach()).pow(2) * weights
prios = loss + 1e-5
loss  = loss.mean()

loss.backward()
replay_buffer.update_priorities(indices, prios.data.cpu().numpy())
optimizer.step()

return loss


(V1) As you can see, the weights are used to calculate the loss and the prios. The loss is the mean squared error including the weights?

loss  = (q_value - expected_q_value.detach()).pow(2) * weights
prios = loss + 1e-5
loss  = loss.mean()


(V2) In "Hands-On Reinforcement Learning for Games: Implementing self-learning agents" on page 176 they use ONE value for ALL prios, coming from the first pow(2).mean():

loss  = (q_value - expected_q_value.detach()).pow(2).mean()
prios = loss + 1e-5
loss  = loss.mean()


(v3) Here loss is the mean squared error, prios are the raw errors:

terr  = (current_Q - target_Q.detach())
prios = terr + 1e-5
loss  = terr.pow(2).mean()


(v4) Here loss is the mean squared error, prios are the squared errors:

terr  = (current_Q - target_Q.detach())
prios = terr.pow(2) + 1e-5
loss  = terr.pow(2).mean()


(v5) Here loss is the mean squared error, prios are the absolute errors:

terr  = (current_Q - target_Q.detach())
prios = torch.abs(terr) + 1e-5
loss  = terr.pow(2).mean()


(v6) Here loss is the mean squared error, prios are the squared absolute errors:

terr  = (current_Q - target_Q.detach())
prios = torch.abs(terr).pow(2) + 1e-5
loss  = terr.pow(2).mean()


Which version is correct and why?