0
$\begingroup$

I'm trying to implement Prioritized Experience Replay (PER) for a Deep-Q Network (DQN) as described in this paper: https://arxiv.org/pdf/1511.05952.pdf. In section 3.4 it discusses how priority sampling adds a bias as the samples are obviously no longer uniform.

The DQN paper from Nature (https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf) clips the error term; I'm using Huber Loss in my implementation to perform error clipping, which is in line with what I've seen in other DQN projects. Here's my Huber Loss implementation (Keras, using a $\delta$ parameter of 1.0):

import tensorflow as tf

def huber_loss(y_true, y_pred):
  error = tf.keras.backend.abs(y_true - y_pred)
  cond  = error <= 1.0

  squared_loss = 0.5 * tf.keras.backend.square(error)
  linear_loss  = error - 0.5

  return tf.keras.backend.mean(
    tf.keras.backend.switch(cond, squared_loss, linear_loss))

I'd like to update this loss function such that the importance sampling (IS) weights are folded in, but I'm not sure where to multiply the IS weights. I.e. is the following correct?

def huber_loss(y_true, y_pred, is_weights):
  error = tf.keras.backend.abs((y_true - y_pred) * is_weights)
  cond  = error <= 1.0

  squared_loss = 0.5 * tf.keras.backend.square(error)
  linear_loss  = error - 0.5

  return tf.keras.backend.mean(
    tf.keras.backend.switch(cond, squared_loss, linear_loss))

Or possibly this?

def huber_loss(y_true, y_pred, is_weights):
  error = tf.keras.backend.abs(y_true - y_pred)
  cond  = error <= 1.0

  squared_loss = 0.5 * tf.keras.backend.square(error)
  linear_loss  = error - 0.5

  return tf.keras.backend.mean(
    is_weights * tf.keras.backend.switch(cond, squared_loss, linear_loss))

Or maybe neither is correct? To me, the first seems like the correct way to go about it: errors exceeding 1, when scaled down by the IS weights, may fall within the squared_loss.

OpenAI Baselines is one of the only implementations I can find that incorporates IS weights, and they use the second version. That said, OpenAI Baselines' DDQN with PER performs quite poorly: in it's current state it takes over 30GB of memory to run, and it doesn't come anywhere near the numbers reported in the Nature Paper.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.