1
$\begingroup$

I finally developed a Game Bot that learns how to play the videogame Snake with Deep Q-Learning. I tried with different neural networks and hyper-parameters, and I found a working set-up, for a specific set of rewards.
The problem is is: when I reward the agent for going in the right direction - positive rewards in case the coordinates of the agent increase or decrease accordingly to the coordinates of the food - the agent learns pretty fast, obtaining really high scores. When I don't reward the agent for that, but only negative rewards for dying and positive for eating the food, the agent does not learn. The state takes into account if there's any danger in proximity, if the food is up, down, right or left and if the agent is moving up, down, right or left.
Here's the question: is rewarding the agent for going into the right direction a "correct approach" in Reinforcement Learning? Or it's seen as cheating, cause the system needs to learn that by itself? Is passing the coordinates of the food as state an other way of "cheating"?

$\endgroup$
1
$\begingroup$

Here's the question: is rewarding the agent for going into the right direction a "correct approach" in Reinforcement Learning?

It depends on what you are hoping the agent is capable of learning by itself. This is an issue for you here, because you have a "toy" problem where you can control a lot more of the environment and alter the meaning of what it means to win.

In general, then yes this is "cheating", at least in terms of claiming to have written an RL agent that solves the game. The academically ideal basic RL agent is rewarded by the gain of something meaningful in the context of the problem being solved, and is not helped by interim rewards. In a game of snake, and any other arcade-style game, it should really be the official points scored in the game and nothing else.

Is passing the coordinates of the food as state an other way of "cheating"?

Again it depends on what you expect the agent to learn from. If, in your target production environment, this data was easy to obtain, and you intended to use it to write a game bot working from the trained policy, then this is fine. There is no requirement that you do one thing or another if you have a practical problem to solve.

However, learning from a pixel-only state, as in the DQN original papers, is of academic interest, because that is a generic state representation that applies to many problems, whilst the distance from the snake to food is a specific feature that you have engineered that makes learning easier in a smaller set of games.

The main issue here is again that your goal is not really to put a "snake bot" into a production system, but to learn how RL works. RL is tricky, and often doesn't work as well as you expect - or at all, for many combinations of algorithm and problem.

It is worth reading this article: Deep Reinforcement Learning Doesn't Work Yet - it may put disappointing results from basic DQN into perspective.

I would encourage you to strip back your Snake problem to remove "helpful" rewards and state, and instead look into extensions to the core DQN algorithm, or different learning agents such as A3C.

$\endgroup$
  • $\begingroup$ Interesting article, and thank you as always for your detail answer. Exactly as you said, I am only doing this for learning. I am disappointed, because it's supposed to be a fairly easy task with a pretty straightforward relationship between state and action, and still the system does not learn. $\endgroup$ – Mauro Comi Sep 20 '18 at 10:20
  • $\begingroup$ @MauroComi: It might be worth reviewing your state representation or other details of how you have implemented basic DQN in case there is a hidden problem. $\endgroup$ – Neil Slater Sep 20 '18 at 10:24
  • $\begingroup$ I agree with Neil here. A couple of comments. Change your DQN to learn from pixels. Some tasks are impossible if there is not some kind of reward shaping (e.g. hierarchical tasks). Get a sense of your state space and intuition of your reward is really sparse. If a simple Q-learning can explore enough can solve your problem. If DQN cannot then something you are doing wrong. Stick to basic DQN till you know it well and test it in different tasks to get a sense of what it does and how inputs affect the outputs. PGs are way more complicated and you will have no clue why it fails/succeeds. $\endgroup$ – Constantinos Sep 21 '18 at 3:14

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.