My question deals with the algorithm described in the paper: Reinforcement Learning for Optimized Trade Execution
This paper uses reinforcement learning technique to deal with the problem of optimized trade execution. They divide the data into episodes, and then apply (on page 4 in the link) the following update rule (to the cost function) and algorithm to find an optimal policy:
T is the total time units,
I is the volume,
L is the possible number of actions, x represents the state, and
c represents the cost function, and
c_im is the immediate reward at a certain state and a certain action.
n is the number of times that the state-action pair were visited)
Here are my questions:
If I understand correctly, the algorithm is basically a dynamic programming, when we move backwards in time.
Why do we need
nin the cost function update rule. Aren't we visiting each state exactly once?
If I understand correctly, we should run this algorithm on every episode (in the experiment in the paper they had 45000 episodes). In such case, how do we combine the results from all the episodes? That is, each episode provides an optimal policy. How do we combine all these policies to one final policy?