0
$\begingroup$

Let's say, you want to apply reinforcement learning on a simple 2D game. (ex : super mario)

The easy way is of course to retrieve an abstraction of the environnment, per example using gym and/or an open-source implementation of the game.

But if it's not available, I think about integrating convolutionnal layers over pixels as inputs of the RL agent.

We could of course split the task in two : featurization of the images and then reinforcement learning, we probably would need some supervision over the images (which can be problematic since we have no abstraction of the environment).

Is it a feasible approach to combine learning a featurization of the image data and learning a game policy at the same time ?

Improve this question
$\endgroup$
2
  • $\begingroup$ It is not clear what you are asking - do you want to perform RL on a screen image (and your suggestion of object recognition is your idea of a solution), or to perform RL/object recognition together somehow? Have you heard of or read the original DQN paper (it is well known and commonly referenced in RL introductions, including Sutton & Barto)? The environment was Atari games, and the agent performed Q learning from screen pixels (using a CNN). It did not also perform object recognition, although inspection shows that it learns important objects indirectly. $\endgroup$
    – Neil Slater
    Commented Apr 15, 2022 at 17:42
  • $\begingroup$ Hello Neil, thanks for your answer. My question is about performing RL over screen image. Instead of object recognition, I should correct to image featurization. The paper shows a method that is completly reusable on my case. $\endgroup$
    – klegoff
    Commented Apr 15, 2022 at 17:53

1 Answer 1

Reset to default
0
$\begingroup$

Yes, it is possible to use convolutional layers in a reinforcement learning (RL) agent approximation function for action values (e.g. Q learning) or for policies (e.g. REINFORFCE).

In fact, any learning system capable of online learning of functions from example inputs and outputs will work with RL. The RL component will generate the examples to learn by taking actions in the environment or in simulation, and calculating some value such as the expected return. These examples are drawn from different distributions as the agent becomes better at the task, which is why online learning is important - the agent must forget the values associated with earlier experiences and replace them with new values as it improves its performance. Neural networks work for online learning by default, unless you make changes to them to prevent that.

That means you are not restricted to simple feed-forward networks. You can use CNNs, RNNs and other flavours of neural network provided you design them to output your value function or policy. Which will be best to use depends on the nature of the environment and your input signals. CNNs are a good choice whenever there is a structured arrangement of similar inputs - that includes image data, also many board games.

If you have not already, you may want to get hold of a (free PDF) copy of Reinforcement Learning: An Introduction. In chapter 16, section 16.5 the authors explain the original DQN project which learned how to play video games, including a discussion of the neural network architecture and pre-processing used. This is nowadays a well-known result in the RL community, you will find discussions, examples and implementations of it in many places.

One of the original researchers on DQN, David Silver, has published a lecture series on RL, with videos available on YouTube. He is also associated with DeepMind's Alpha Go project, which is another example RL system that uses CNN architecture internally.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.