0
$\begingroup$

I've started reading some literature about reinforcement learning and I can't understand what is the result of the application of RL. I'll be more specific: let's have a time series problem in continuous state space, finite numbers of actions and a linear approximator of the policy function. So I follow an algorithm to find the best policy, that is, in this specific case, the optimal values of the weights of the linear function I've considered. Now my doubt is here: the so-called best policy is the one found in the process of applying the algorithm or I have to take the final optimal values and, for each period, I have to use them to find which action maximise the action-value function? In other words, the result of RL is a classic function to (re)apply at each time step, as if it was a regression? I think the answer of this question is No, but I would appreciate if someone can confirm this.

(to better explain what I meant with "policy found in the process of applying the algorithm" let's consider this stupid consideration: the best policy also include those time steps of exploration)

| improve this question | |
$\endgroup$
0
$\begingroup$

Is the result of RL is a classic function to (re)apply at each time step?
In some manner yes, when using RL to find the best policy you end up with a policy that can be described as a function(classic or not) from possible states to possible actions.

As if it was a regression?
No, regression algorithms 'solves' the function between feature space and target space.
In RL both of these spaces have no(very different) meanings.
In addition, RL algorithms take into consideration multi-step predictions(state transitions + rewards), which is not very straight-forward in regression problems.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Yes of course, maybe I didnt' explain very well my problem (which at the end I think will be trivial). Let's try a different approch. If I have a time series problem whenever time passes you receive a new state: so do I have to keep updating the weights of the action-value function approximators and then use the updated function to find out what's the best action at that time? $\endgroup$ – aandre_90 May 6 at 14:22
  • $\begingroup$ Are you referring to online-learning? learning a new function every time new data becomes available? if not, from what I know, multi-step prediction in time series modeling uses the same function with different values. One of the big difference between time-series and 'regular' regression is the independence assumption of Y's(or errors) $\endgroup$ – yoav_aaa May 6 at 15:12
  • $\begingroup$ Yes, online learning (and the temporal difference method) $\endgroup$ – aandre_90 May 6 at 15:21
  • $\begingroup$ Sorry, I think im add more confusion. I hope someone else could help. $\endgroup$ – yoav_aaa May 6 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.