# Deep Reinforcement Learning - mean Q as an evaluation metric

I'm tuning a deep learning model for a learner of Space Invaders game (image below). The state is defined as relative eucledian distance between the player and the enemies + relative distance between the player and 6 closest enemy lasers normalized by the window height (if the player's position is $$(x_p,y_p)$$ and an enemy's position is $$(x_e,y_e)$$, the relative euclidian distance is $$\frac{\sqrt{(x_p-x_e)^2+(y_p-y_e)^2}}{HEIGHT}$$ and HEIGHT is the window height). Hence the observation space dimension is (10+6), which results in an input of my deep neural network of 16 units.

My agent doesn't seem to learn (reward function doesn't increase) and I thought I'd check the mean Q values, which are the output of my main deep neural network, and, instead of increasing, I've remarked that the mean Q values stabilizes (as in figure below) instead of increasing. I've modified many tuning parameters (batch size, neural net architecture and parameters...) but I still have same problem. Any idea why the mean Q values wouldn't increase ?

Here are some results about the learner:

• It is not clear to me what your question is. Are you asking how/why to use mean Q as an evaluation metric (your title suggests this) or asking why your agent is not learning? I think it would help if you showed a picture plus explained your state representation in more detail. I'd like to know what distance metric you are using (Euclidean?), what typical values are, and whether you are normalising the data before passing it to the neural network. If you are interested in mean Q metric, please also supply your reward scheme. Please use edit to add those details to the question – Neil Slater Aug 19 '20 at 8:30
• Thanks for the answer. I'm actually wondering why the mean Q is not increasing in order to understand why the agent is not learning. The edit is done! :) – Yassine Aug 19 '20 at 19:22
• What is your agent's and NN's design/architecture? Is it based on DQN? – Sammy Aug 19 '20 at 19:34
• It's 16 > 100 > 100 > 4 (for 4 actions: Left, right, stay still and shoot). I've also tried a similar architecture to the convolutional NN of Space Invaders of Atari games (nihit.github.io/resources/spaceinvaders.pdf) and other architectures I've found on the net on similar games. Doesn't really change the results. And yes it's based on DQN – Yassine Aug 19 '20 at 19:52

I think your main problem is use of relative distance as the core feature. It has two major weaknesses:

• The distance to an object does not give the direction to the object. The best action choices are all critically dependent on direction. For example an enemy laser bolt 0.1 units directly above the player is an immediate danger requiring evasive action, whilst one 0.1 units to the left or right is not a danger and about to leave the game window. Your feature of relative distance does not distinguish between those scenarios, but it is a critical difference.

• Slightly less important, but the raw distance does not capture any sense of movement. If enemies move consistently turn by turn, but not always in the exact same direction or same speed, then their velocities should also be part of the state.

One way you could improve the features is to add a velocity component for each item, showing how quickly it is approaching or receding from the player. This might help a little, but my feeling is that you need more data than distance and speed.

I think you should use normalised $$x, y$$ position as features for each item being tracked, plus normalised velocity $$dx, dy$$ for any object type that can change direction (if enemy lasers are always falling straight down you might not need anything for those).

• If the window edges are important, you should include at least the relative $$x$$ of one of them, so the agent knows its absolute position on screen and how much space it has to maneuver. This is true whether the player is blocked from moving further left or right, or whether the player "wraps around" to the other side of the screen. Both types of effect will significantly affect how the game plays near the screen edge.