As I understand the question - a lot of this is about simple vision processing.
If you already had a access to the position of the ball and the position of the two paddles, one has almost everything required to be able to play the game - the rest is machine learning.
In the simplest sense the agent gets a sequence of tuples of 4 numbers, two for the ball and one each for the paddles. And it has to decide whether to move its paddle up or down. Assuming we are not trying to get the agent to double guess its opponent, this decision could be made in an entirely stateless manner. One just needs to decide up/down given the four numbers.
I assume we are talking about classic pong?
How to image process this?
I am assuming you are not intending for the agent to learn the vision processing? That would be a different discussion.
This problem is very simple in terms of modern vision processing.
each paddles is a solid block that is the only thing inside a rectangle on the screen.
the ball in play is a simple square inside an otherwise empty region of the screen. (Mild problem if as it crosses the mid line).
There may well be faster ways to do it - but the simplest way is to scan a line along the middle of each vertical rectangle looking for a section that has a high average intensity.
Similarly for the ball - there are different search patterns, but one could scan down a collection of vertical lines looking for a similar section of high average intensity.
What I am describing above is essentially a simplified version of finding a shape on the screen by using convolution/correlation between the shape you want and the screen.
If you want to read the block-characters on the screen, like the score, then a similar approach will make it possible to tell the difference between each of the characters. Worst case - you know exactly where they are to just look for correlation between a template of each character and the image on the screen.
I am assuming that you are working from a clean bit-mapped screen shot, rather than, for example, a camera pointed at the screen of the game.