1
$\begingroup$

I've started reading some literature about reinforcement learning and I can't understand what is the result of the application of RL. I'll be more specific: let's have a time series problem in continuous state space, finite numbers of actions, and a linear approximator of the policy function. So I follow an algorithm to find the best policy, that is, in this specific case, the optimal values of the weights of the linear function I've considered. Now my doubt is here: the so-called best policy is the one found in the process of applying the algorithm or I have to take the final optimal values and, for each period, I have to use them to find which action maximizes the action-value function? In other words, the result of RL is a classic function to (re)apply at each time step as if it was a regression? I think the answer to this question is No, but I would appreciate it if someone can confirm this.

(to better explain what I meant with "policy found in the process of applying the algorithm" let's consider this stupid consideration: the best policy also include those time steps of exploration)

$\endgroup$

2 Answers 2

1
$\begingroup$

So there are a few things you seem to be confused about. Short answer is no.

In Reinforcement Learning (RL), the goal is to learn a policy for taking actions in a Markov Decision Process (MDP) to maximize a reward. If your problem can be described as a Markov Decision Process, then RL may be a good solution. Theoretical results show that with proper annealing, a linear policy, continuous state space, finite actions, the "Q-Learning" RL algorithm will converge to an optimal linear policy, where Q-Learning learns a function that maps from (state, action) to expected discounted sum of reward.

A Markov Decision Process is easiest to think of as a graph. In an "episode", we have an initial state (node of the graph), then at each step we transition (along an edge) to another state (node) until we reach (or possibly never reach) a terminal state ending the episode. During each step we also choose an action and receive a reward. What state you transition to after each step is random, but the "transition probability" is a function of your current state and chosen action [$P(s')=f(s,a)$], and our reward is random but the probability is a function of our current state, action, and resulting state [$P(r)=f(s,a,s')$]. Our goal is to maximize the expected sum of this reward (discounted sum technically). In effect, we're randomly bouncing around this graph from node to node, taking actions that influence our destination node, and collecting rewards. In your case, the graph isn't a necessary abstraction and instead our state is a continuous vector.

Q-Learning (and RL algorithms in general) learn by playing repeated episodes in our MDP, learning to optimize the discounted sum of rewards. After each episode Q-Learning updates a learned function that maps from (state, action) to expected discounted sum of reward. Algorithms trades off "exploiting" patterns they've learned for reward, and "exploring" new (state, action) pairs. So the algorithm is not necessarily maximizing the reward while training.

So to answer your question. Q-learning does not learn within the episode. It updates the learned function after each episode, eventually converging to your final policy. That final policy is what you use in your application. That policy is a function that maps from observed state and action to expected reward. This works as long as there is no "hidden" or "unobserved" information that changes within the episode or from episode to episode. If there is "hidden" information, then a RL/MDP may be a poor fit. It may instead be a "POMDP", which require other tools than RL to solve. Further, if you cannot reset the environment and run multiple episodes, then RL/POMDP/MDP will be a poor fit. That being said, plenty of people have applied RL successfully to problems that don't fit these rules (e.g. multi-agent RL). So if your problem doesn't fit, this is more of a warning than a rule.

$\endgroup$
0
$\begingroup$

Is the result of RL is a classic function to (re)apply at each time step?
In some manner yes, when using RL to find the best policy you end up with a policy that can be described as a function(classic or not) from possible states to possible actions.

As if it was a regression?
No, regression algorithms 'solves' the function between feature space and target space.
In RL both of these spaces have no(very different) meanings.
In addition, RL algorithms take into consideration multi-step predictions(state transitions + rewards), which is not very straight-forward in regression problems.

$\endgroup$
4
  • $\begingroup$ Yes of course, maybe I didnt' explain very well my problem (which at the end I think will be trivial). Let's try a different approch. If I have a time series problem whenever time passes you receive a new state: so do I have to keep updating the weights of the action-value function approximators and then use the updated function to find out what's the best action at that time? $\endgroup$
    – unter_983
    May 6, 2020 at 14:22
  • $\begingroup$ Are you referring to online-learning? learning a new function every time new data becomes available? if not, from what I know, multi-step prediction in time series modeling uses the same function with different values. One of the big difference between time-series and 'regular' regression is the independence assumption of Y's(or errors) $\endgroup$
    – yoav_aaa
    May 6, 2020 at 15:12
  • $\begingroup$ Yes, online learning (and the temporal difference method) $\endgroup$
    – unter_983
    May 6, 2020 at 15:21
  • $\begingroup$ Sorry, I think im add more confusion. I hope someone else could help. $\endgroup$
    – yoav_aaa
    May 6, 2020 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.