Combining Neural Network with Reinforcement Learning in a Continuous Space

Question

I'm trying to learn how to do reinforcement learning on my own and I am not sure how to implement a neural network for a specific problem.

The game goes on for roughly 1 million steps. At each step, I have 36 continuous features available (unknown rules), and 4 actions to choose from. Afterwards, the environment will tell me what the 36 features are for the next step, and what my score is. I've been creating and looking at the data, and it sometimes only becomes clear a move was good/bad a lot later (10 moves or so).

So there are 2 things involved.

1) I have to "learn" how, given an action, the world would change from pre_state to post_state, and

2) I have to learn to optimize the reward given the state and choosing from 4 actions.

So I was thinking to just record state_before, state_after, reward of a lot of random moves. I could perhaps just use 36 * 10 of the last moves as predictors. But then again, perhaps I should only be interested in the difference between states?

The problem lies in the fact that I've looked at Markov Decision Processes, but they assume a discrete search space (whereas here it is continuous).

Any help would be useful in trying to understand what layers should be involved to solve this, and perhaps what the most logical way would be to sample data.

I really hope people can point me in the right direction. I'm willing to use any neural networks framework in Python basically.

Answer 1

I might be a bit late for answering but hope it helps!I assume that you are familiar with RL so I will omit lots of details (please if you are still interested comment so I can help you).

Neural Networks and RL: You have two options. The first one is to use a network which you will have as input your feature vectors (states) and output probability of each action. This is called a policy network and you can find a very detailed tutorial with Python Code in order to implement it by A. Karpathy. Your second option is to use the Q-Network approach. Your input will be again the same but the output will be values of your Q function for each action you have ($Q(a_i)$). You will use the Q-learning equation $Q(s_t,a_t)=Q(s_t,a_t) +\alpha[r_t+\gamma \max _a'Q(s_{t+1},a')-Q(s_t,a_t)]$. The details of the implementation can be found in the paper of V. Mnih. Also do not worry about the delayed rewards as the discount factor $\gamma will "help" your agent to be affected by the future rewards.

In order to calculate your states, I would suggest you to create a simulation of the environment and a step function. You don't mention what kind of game you are dealing with but the general idea is that the step function will take as input your current state and current action and output the next state and reward (don't mind about the continuous space of your features as you can discretize it - you can use Kalman filters or other models to have a better state estimation as well).

My advice would be to choose your approach (Policy net or Q-net) and read the blog or the paper, create a simulation of the environment and a step function for your game. You can find tons of implementations of the Deep Q-net although I would suggest you to start with a very simple network so you don't get in trouble by tuning the Deep net.