1
$\begingroup$

I'm tuning a deep learning model for a learner of Space Invaders game (image below). The state is defined as relative eucledian distance between the player and the enemies + relative distance between the player and 6 closest enemy lasers normalized by the window height (if the player's position is $(x_p,y_p)$ and an enemy's position is $(x_e,y_e)$, the relative euclidian distance is $\frac{\sqrt{(x_p-x_e)^2+(y_p-y_e)^2}}{HEIGHT}$ and HEIGHT is the window height). Hence the observation space dimension is (10+6), which results in an input of my deep neural network of 16 units.

enter image description here My agent doesn't seem to learn (reward function doesn't increase) and I thought I'd check the mean Q values, which are the output of my main deep neural network, and, instead of increasing, I've remarked that the mean Q values stabilizes (as in figure below) instead of increasing. I've modified many tuning parameters (batch size, neural net architecture and parameters...) but I still have same problem. Any idea why the mean Q values wouldn't increase ?

enter image description here

Here are some results about the learner: enter image description here

$\endgroup$
4
  • $\begingroup$ It is not clear to me what your question is. Are you asking how/why to use mean Q as an evaluation metric (your title suggests this) or asking why your agent is not learning? I think it would help if you showed a picture plus explained your state representation in more detail. I'd like to know what distance metric you are using (Euclidean?), what typical values are, and whether you are normalising the data before passing it to the neural network. If you are interested in mean Q metric, please also supply your reward scheme. Please use edit to add those details to the question $\endgroup$ – Neil Slater Aug 19 '20 at 8:30
  • $\begingroup$ Thanks for the answer. I'm actually wondering why the mean Q is not increasing in order to understand why the agent is not learning. The edit is done! :) $\endgroup$ – Yassine Aug 19 '20 at 19:22
  • $\begingroup$ What is your agent's and NN's design/architecture? Is it based on DQN? $\endgroup$ – Sammy Aug 19 '20 at 19:34
  • $\begingroup$ It's 16 > 100 > 100 > 4 (for 4 actions: Left, right, stay still and shoot). I've also tried a similar architecture to the convolutional NN of Space Invaders of Atari games (nihit.github.io/resources/spaceinvaders.pdf) and other architectures I've found on the net on similar games. Doesn't really change the results. And yes it's based on DQN $\endgroup$ – Yassine Aug 19 '20 at 19:52
0
$\begingroup$

I think your main problem is use of relative distance as the core feature. It has two major weaknesses:

  • The distance to an object does not give the direction to the object. The best action choices are all critically dependent on direction. For example an enemy laser bolt 0.1 units directly above the player is an immediate danger requiring evasive action, whilst one 0.1 units to the left or right is not a danger and about to leave the game window. Your feature of relative distance does not distinguish between those scenarios, but it is a critical difference.

  • Slightly less important, but the raw distance does not capture any sense of movement. If enemies move consistently turn by turn, but not always in the exact same direction or same speed, then their velocities should also be part of the state.

One way you could improve the features is to add a velocity component for each item, showing how quickly it is approaching or receding from the player. This might help a little, but my feeling is that you need more data than distance and speed.

I think you should use normalised $x, y$ position as features for each item being tracked, plus normalised velocity $dx, dy$ for any object type that can change direction (if enemy lasers are always falling straight down you might not need anything for those).

In addition:

  • If the window edges are important, you should include at least the relative $x$ of one of them, so the agent knows its absolute position on screen and how much space it has to maneuver. This is true whether the player is blocked from moving further left or right, or whether the player "wraps around" to the other side of the screen. Both types of effect will significantly affect how the game plays near the screen edge.

  • In order to track predicted value, you need to track location of player missiles. It is not enough to just let the agent predict when it is best to fire - in order to accurately track a value function it needs to "see" whether the missile it fired some time steps ago is likely to hit or miss a target.

  • For both enemy lasers and player missiles, it is OK to filter and sort the data by some criteria (such as distance to player). As long as this is consistent it may even help a lot to have such pre-processing.

$\endgroup$
8
  • $\begingroup$ Thank you very much. I'll to look it up more in depth & modify my code and I'll let you know! $\endgroup$ – Yassine Aug 20 '20 at 20:43
  • $\begingroup$ I understand the fact that the relative distance alone as a feature is not enough. I'm adding a sign to the relative distances to distinguish the enemies/laser that are on the left or right of the player + the player's position (x,y) + player's laser position normalized as you suggested. The problem is that the number of the player's missiles is not known so I guess I'll just track a fixed number of them. However, I don't think the velocity component for each item would be important since the enemies are moving at the same velocity. $\endgroup$ – Yassine Aug 21 '20 at 21:09
  • $\begingroup$ @Yassine - if the enemies can switch direction, then their velocity can change. That's the difference between speed and velocity. Speed is a magnitude only, and velocity is magnitude and direction. youtube.com/watch?v=bOIe0DIMbI8 $\endgroup$ – Neil Slater Aug 22 '20 at 9:01
  • $\begingroup$ Yes I see, "Vector" explained it very clearly too, thanks. Even after the modification of the observation space in order to consider all the above features in my previous comment, the curves (mainly mean Q, reward) don't seem to change very much: I still have a stabilization of mean Q between 0.5 and 0.75 and reward fluctuating around 2300 just like what I posted. I'll do a bit more of "feature engineering" & modify some other parameters and let you know if it changes! Thank you :) $\endgroup$ – Yassine Aug 22 '20 at 12:31
  • $\begingroup$ NeilSlater, I want to publish an article on Medium and I want to thank you for your help. I don't know how to mention you though ? do you have a blog or something ? Thanks $\endgroup$ – Yassine Nov 1 '20 at 10:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.