# Combining Neural Network with Reinforcement Learning in a Continuous Space

I'm trying to learn how to do reinforcement learning on my own and I am not sure how to implement a neural network for a specific problem.

The game goes on for roughly 1 million steps. At each step, I have 36 continuous features available (unknown rules), and 4 actions to choose from. Afterwards, the environment will tell me what the 36 features are for the next step, and what my score is. I've been creating and looking at the data, and it sometimes only becomes clear a move was good/bad a lot later (10 moves or so).

So there are 2 things involved.

1) I have to "learn" how, given an action, the world would change from pre_state to post_state, and

2) I have to learn to optimize the reward given the state and choosing from 4 actions.

So I was thinking to just record state_before, state_after, reward of a lot of random moves. I could perhaps just use 36 * 10 of the last moves as predictors. But then again, perhaps I should only be interested in the difference between states?

The problem lies in the fact that I've looked at Markov Decision Processes, but they assume a discrete search space (whereas here it is continuous).

Any help would be useful in trying to understand what layers should be involved to solve this, and perhaps what the most logical way would be to sample data.

I really hope people can point me in the right direction. I'm willing to use any neural networks framework in Python basically.

Neural Networks and RL: You have two options. The first one is to use a network which you will have as input your feature vectors (states) and output probability of each action. This is called a policy network and you can find a very detailed tutorial with Python Code in order to implement it by A. Karpathy. Your second option is to use the Q-Network approach. Your input will be again the same but the output will be values of your Q function for each action you have ($Q(a_i)$). You will use the Q-learning equation $Q(s_t,a_t)=Q(s_t,a_t) +\alpha[r_t+\gamma \max _a'Q(s_{t+1},a')-Q(s_t,a_t)]$. The details of the implementation can be found in the paper of V. Mnih. Also do not worry about the delayed rewards as the discount factor \$\gamma will "help" your agent to be affected by the future rewards.