2
$\begingroup$

I'm working on below reinforcement learning problem: I have bottle of fix capacity (say 5 liters). At the bottom of bottle there is cock to remove water. The distribution of removal of water is not fixed. we can remove any amount of water from bottle, i.e. any continuous value between [0, 5].

At the top of the bottle one tap is mounted to fill water in the bottle. RL agent can fill [0, 1, 2, 3, 4] liters in the bottle. Initial bottle level is any value between [0, 5].

I want to train the agent in this environment to get optimal sequence of actions such that bottle will not get empty and overflow which implies continuous supply of water demand.

Action space = [0, 1, 2, 3, 4] Discrete Space

Observation Space = [0, Capacity of Bottle] i.e. [0, 5] Continuous Space

Reward logic = if bottle empty due to action give negative rewards; if bottle overflow due to action give negative rewards

I have decided to use python to create an environment.

I'm new to RL. I'm not aware about for which condition I have to give the rewards. Is my reward logic is correct or I have to change it?

$\endgroup$
1
  • $\begingroup$ The site does not give out generic guidance and help. If you could focus the question on a single part of your problem that you immediately face, then it can be answered in one go. If that doesn't take you from start to end in one go, then you can ask a different question when you next get stuck. The next step for you should be to remove the last paragraph (where you ask "Can some help me/ guide me to write the environment") and replace with something you really need to know when you are doing the work (e.g. "What methods do I need to represent this environment to an agent?") $\endgroup$ – Neil Slater Aug 16 '18 at 20:12
0
$\begingroup$

I'm not aware about for which condition I have to give the rewards. Is my reward logic is correct or I have to change it?

The reward structure in reinforcement learning is a little bit flexible, provided it expresses the goal of your agent.

In your case, you want to avoid two bad scenarios in a continuous problem. There is also a good scenario - providing enough water - which overlaps with the negative consequences of not providing water on demand, but may be subtly different.

Your suggestion of granting a negative reward for spilled water, and a negative reward for unavailable water seems quite reasonable. You have a few choices within that that could make differences to the behaviour of an optimal agent. Given your problem description so far, I think these are a free choice for you:

  • You can decide on a fixed cost for an error e.g. $-1$ reward, or base the cost on the amount of water spilled or missing when required.

  • You can weight rewards for water spilled differently to water missing when required. This is the sort of thing that might be a business decision, based on different costs for the different kind of errors. If spillage costs are low, demand is high and impact of failed distribution is high, then the optimal policy may involve over-filling the tank and spilling water deliberately.

  • You can have a positive reward for the amount of water supplied. This is similar to having a negative reward for failed supply, but again might be weighted differently in a business scenario. If you are supplying water to paying clients, then you would care about profits here and have 3 types of reward covering the profits from selling water, the loss from wasted water and clean-up costs, and (the maybe more nebulous, but still estimated) loss from failing to meet customer demand when supply is not available.

You should bear in mind, that combined with the behaviour of the environment, these decisions may change what the optimal policy is. Until you define these values, there is no single optimal policy. As your problem is simple, there may be a lot of overlap between optimal policies, but choices you make in reward functions and how the environment behaves will result in differences perhaps in how willing the agent is to risk over-filling in some cases.

As the problem is continuous, the long-term reward that you want to optimise could become infinite. To avoid this, and make the problem tractable, you need to make one of the following choices:

  • Use a discount factor, $\gamma$, which can be from $0$ to $1$ (but not actually 1 for a continuous problem)

  • Formulate your goal as maximising average reward.

In your case I suggest using a highish discount factor, e.g. $\gamma = 0.99$, as most of the RL literature for value-based agents uses discount factors.

Finally, you may have an issue here:

The distribution of removal of water is not fixed. we can remove any amount of water from bottle, i.e. any continuous value between [0, 5]

RL depends on the environment behaving as a Markov Decision Process. Which means the theory works best if there is a fixed distribution here. That could be a flat distribution in $[0,5]$, or it could be drawn from a complex real-world scenario, but the assumption from the theory is that the distribution of changes to state is a function of most recent state and action plus a stochastic element.

When you simulate your problem in order to train the agent, you will have to use an actual distribution. The agent will then optimise against the scenarios it experiences. Depending on what you simulate, it may do poorly in other situations. However, provided you don't take this to extremes, the agent should learn to optimise over a broad range of states, and many reasonable distributions of the removal of water.

$\endgroup$
3
  • $\begingroup$ Thanks Neil for detail answer. I'm already using the fixed same cost -2 for both the errors. As per your suggestion I'll try the cost on the amount of water spilled or missing when required. Currently I'm using DQN for this problem with below hyper-parameter settings: memory = 2000; discount rate( gamma) = 0.95; learning rate = 0.001; exploration rate=1 (at beginning); epsilon decay = 0.99; epsilon min = 0.01. $\endgroup$ – Krishna Nevase Aug 17 '18 at 8:35
  • $\begingroup$ @KrishnaNevase: No problem. Your rewards should be fine to produce a trainable agent. Changing the rewards, as long as they are valid, will just change what the optimal policies are. DQN has a few hyperparameters to tune, in the design of the neural network. You could avoid that and simply discretise your state, making the problem much simpler if your goal is to learn RL. However, if your goal is to learn DQN as well, may as well stick with what you have. $\endgroup$ – Neil Slater Aug 17 '18 at 8:45
  • $\begingroup$ Thanks. Added another question related to this RL problem new question $\endgroup$ – Krishna Nevase Aug 17 '18 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.