# Reinforcement learning, pendulum python

I'm having trouble finding a good reward function for the pendulum problem, the function I'm using: $-x^2 - 0.25*(\text{xdot}^2)$ which is the quadratic error from the top. with $x$ representing the current location of the pendulum and $\text{xdot}$ the angular velocity.

It takes a lot of time with this function and sometimes doesn't work. Any one have some other suggestions? I've been looking in google but didn't find anything i could use

You could use the same reward function that Openai's Inverted Pendulum is using:

$costs=-(\Delta_{2\pi}\theta)^2 - 0.1(\dot{\theta})^2 - 0.001u^2$

where $(\Delta_{2\pi}\theta)$ is the difference between current and desired angular position performed using modulo $2\pi$. The variable $u$ denotes the torque (the action of your RL agent). The optimal is to be as close to zero costs as it gets.

The idea here is that you have a control problem in which you can come up with a quadratic 'energy' or cost function that tells you the cost of performing an action at EVERY single time step. In this paper (p.33 section 5.2) you can find a detailed description.

I have tested RL algorithms in this objective function and I did not encounter any problems for convergence in both MATLAB and Python. If you still have problems let us know what kind of RL approach you implemented and how you encoded the location of the pendulum.

Hope it helps!

• I think answer is more applicable for OP's problem than mine. Treating the problem as episodic or continuous can be an important factor when deciding on cost function, as per my answer. However, the OP has clarified on my answer that although they have episodes, their other choices mean that an always negative cost function is probably OK. – Neil Slater Feb 15 '17 at 9:41
• Thank you for the comment! I've added the u for the equation. The main problem I'm having is that the agent seems to just rotate the as fast as he can rather than just stop on the top. and i just can't find why he does that I am using episodic approach, and reward with a positive bonus for goal state. – gal isra Feb 15 '17 at 10:38
• Also the update function is Q_[state][action] = Q_[state][action] + learnRate * (reward +discount * best_action(Q[next_state]) - Q[state][action] + bonus) – gal isra Feb 15 '17 at 10:46
• by best_action(Q[next_state]) i meant the value of Q[next_state][best_action[ – gal isra Feb 15 '17 at 11:12
• This problem is a control problem. By using the reward function i posted you will inform the agent, at every single value of your action, what the cost of that action would be and should be enough for the agent's 'guidance'. I think using the additional bonus causes problems especially when the scales of the rewards differ a lot. Also from what I understand, you use a disretisation of your action and state space as I dont see any function approximation. A very nice RL solution for solving this problem in continuous action and state spaces is the Actor-Critic method and can be found here: – Constantinos Feb 15 '17 at 22:17

In reinforcement learning, you should avoid scoring interim results based on heuristics. Unlike supervised learning, or a search algorithm, you are not trying to guide the behaviour, just reward good results. For an inverted pendulum a good result might simply be "has not fallen over so far", although there is nothing inherently wrong with a cost function which expresses cost in terms of minimising differences from an ideal, you do have to take more care with the values used.

Assuming you are using discounting, and continuous (not episodic) approach, then reward can be 0 for "not falling over" and -1 for "it fell over", followed by a re-set/continue. You can check for falling by measuring whether the pendulum has reached some large angle to the vertical (e.g. 45 degrees or more).

For an episodic approach, it is more natural to have +1 "ok" and 0 for the end state "fell over", although the 0/-1 scheme would also work. However, you want to avoid having negative values for any state which is "ok", because that is basically telling the agent to hurry up and end the episode. In your case, ending the episode is bad, so you don't want that.

If you do want to reward "perfection" in your episodic approach, then your formula might work better if you added a positive offset, so that the agent has an incentive to continue the episode if possible. You should choose a value such that recoverable states are positive.

Note that the above analysis applies only to certain episode-based approaches. It depends critically on what you count as an episode, and whether the agent is able to take an action which ends the episode.

• Thank you for your comment! I am using discount but with episodic approach. maybe you have an idea for a reward for that approach? – gal isra Feb 14 '17 at 13:14
• @galisra: I've added that to the answer. With episodic approach, you sometimes have to take care about absolute values, and it looks like that has been a problem for you, because all your rewards were negative - it is much better once things get bad for the agent to end the episode and limit the negative reward. You typically use negative rewards when you want the agent to complete the episode as quickly as possible. – Neil Slater Feb 14 '17 at 13:28
• @galisra: I have revised my answer further. Now I think about it, it is the episodic nature combined with your always negative reward function which is the biggest problem. Your formula may even work by adding a positive offset. For clarification - do your episodes end when the pendulum falls, or always after a fixed number of steps? If after a fixed number of steps, do you continue to score large negative rewards even if the pendulum has fallen? If you use fixed episode length like that, then some of my analysis is wrong. – Neil Slater Feb 14 '17 at 14:15
• I am using a fixed number of steps. also added a poisitive reward for the goal step. so d oyou think its ok to use the approach you suggested with 1, for ok and 0 for fell over? – gal isra Feb 15 '17 at 8:34
• @galisra: Actually, I think Constantinos' answer may be better for your fixed time step episodes, provided you never end an episode early, and always include all rewards. – Neil Slater Feb 15 '17 at 9:38