0
$\begingroup$

What is the correct way to fine-tune a model's (SARSA(0), SARSA(lambda), Q(0), Q(lambda)) parameters, and how can one compare the models?

I read that typically one compares the number of actions needed to achieve the goal, after X episodes. However, my model is slightly different than maze/grid worlds, so I'm not sure on how to do it.

Essentially its a time model, with X-days, and each day an action is selected, the reward on each day depends on the action selected, and the end-state reward depends on the actions selected in the other 249 days. It is an optimization model for a regulatory situation, which implies revising the whole reward every 250 days, based on the performance (which is tied to actions) of the other 249 days.

With that being said, I cannot measure the performance based on the number of actions needed to reach the terminal state, as it will always be reached with the same number of actions.

The models are being trained in a simulation, the optimal policy will be achieved once per model. The state-space is quite large actually, which is one of my concerns about how many epochs (episodes) I should have, its a 250*8*7*3000 space-action (3-state combination, 250,8,7 and 3000 possible actions per state).

Ideally one would compare all possible models - as there are quite a few - with a smaller number of episodes, and the selected ones would then run with the full amount of episodes, for time-reasons.

Any suggestions on how to compare the models? Based on average rewards with the optimal policy's actions?

$\endgroup$
  • $\begingroup$ @NeilSlater First of all, thank you for your replies. For the first section: - Tabular method. They are being trained in a simulation, the optimal policy will be achieved once per model. The state-space is quite large actually, which is one of my concerns about how many epochs (episodes) I should have, its a 250*8*7*3000 space-action (3-state combination, 250,8,7 and 3000 possible actions per state). $\endgroup$ – user4218673 Apr 21 '17 at 8:24
  • $\begingroup$ @NeilSlater for the second post: - Ah so essentially, what it means is that on each state the selected action will contribute towards an índex with X probability. At the end of the 249 days, based on the accumulated índex the final value is calculated. Why am I doing this? Well its just the way the system works, its an optimization model for a regulatory situation, which implies revising the whole reward every 250 days, based on the performance (which is tied to actions) of the other 249 days. $\endgroup$ – user4218673 Apr 21 '17 at 8:28
1
$\begingroup$

As you are building policies in simulation, and can avoid the need to use approximate methods (the state space is small enough to fit in a table in memory), then your goal is to converge on the optimal policy. With correct setup, all the methods you are comparing are guaranteed to converge for your problem, provided the assumptions of reinforcement learning are not broken.

With that being said, I cannot measure the performance based on the number of actions needed to reach the terminal state, as it will always be reached with the same number of actions.

This is just the reward scheme for maze solvers. For those the goal is to solve the maze quickly.

You have a reward scheme. The best policy maximises the expected reward. That is the only measurement that a completed RL output (your policy) cares about. As all your chosen learners should learn the optimal policy, you cannot compare them on this matter. So you maybe need to look at efficiency - how quickly the algorithm converges to optimal policy.

The state-space is quite large actually, which is one of my concerns about how many epochs (episodes) I should have, its a 250*8*7*3000 space-action (3-state combination, 250,8,7 and 3000 possible actions per state).

I would characterise this as a small state space. It will fit into memory as a table. Looping through the 40 million state/action pairs should be pretty fast unless the simulation step is expensive for some reason.

In fact, if you have a simple model for transitions and rewards at each step, I would encourage you to use Value Iteration (a dynamic programming method) instead of the sampling based approaches in SARSA or Q-Learning. To do so, you would need to be able to define code for state transitions and rewards given current state and action. Sometimes though, the simulation is much easier to code than the transitions/rewards model. If that's the case for you, stick with SARSA or Q-Learning.

As there are no consequences to you for bad decisions and low rewards during training stages - learning offline in simulations - then Q-Learning may be preferable as it learns the optimal policy whilst exploring. Compared to SARSA you have to be concerned about how to reduce $\epsilon$ so as to converge on the optimal policy. For Q-Learning you can leave $epsilon$ at a relatively high value (e.g. 0.1) and still learn whilst refining estimates of alternative actions.

However, I am a little concerned about this:

the end-state reward depends on the actions selected in the other 249 days

This is a problem, since it breaks the assumptions of the Markov Decision Process that all the algorithms are based upon. The reward at any step should only be based on current state/action. By doing this, you may invalidate the policy. It definitely would defeat dynamic programming (may prevent it converging) unless you included action history in the state (which would definitely make your state space large). However, it may still work for sampling methods with a high value of $\lambda$, because these are more robust when faced with environments which are not strictly MDPs.

Any suggestions on how to compare the models?

I suggest on CPU time required to learn the optimal model, assuming you intend to repeat this process on different data routinely. The way to find out is to experiment, as convergence will depend on properties of the problem, how fast your simulation code runs, and in your case you are stretching one of the assumptions behind the design of the RL solvers which could make strongly bootstrapping methods (like SARSA(0)) either fail to converge or converge very slowly.

If convergence takes too long, set a cutoff at some time limit (e.g. an hour) and find out expected reward from initial conditions when following the policy learned so far. You could Monte Carlo sample that for a fair comparison (and if results are stochastic even with a deterministic policy, then give plenty of time for 1000s of runs to get a good estimate).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.