Return to Answer

What I am trying to understand is, the reward calculation does not take an action as a parameter. How does it choose an action properly.

The reward function does not choose the action. It gives the immediate consequences for your previous action - and in most cases is a consequence of all previous actions, states and random factors in the environment. Although it should only depend on most recent state, previous action plus a random factor at most.

In your case reward depends directly on the resulting state = how many cars at each junction after taking an action, and not directly on the previous action = which traffic flow to enable. It does still depend on the action indirectly, via however the environment works. But that does not need to be expressed mathematically (as a model of the system) for Q learning to work. Although not stated, the reward probably also depends on a random factor outside of your control = how many cars turn up at each junction.

It is the Q value which eventually ranks the different actions and allows you to select the best action. The value $Q^t(s, a)$ gives you the current best estimate for future rewards (for a continuous problem, such as yours, it is common to have a discount factor, $\gamma$, to give more weight to immediate rewards).

There are multiple ways to select the next action in reinforcement learning, depending on the problem, your learning algorithm etc. In the example pseudo-code you copied, the selection process is using Upper Confidence Bound (or UCB), which is a relatively sophisticated way of balancing exploration of actions that might be better than your current best estimate, versus simply using the "best" action so far. If you always took the "best" action, in many cases this would cause a problem because you would never update your knowledge of what the other actions do.

The standard textbook for all this is Sutton & Barto, Reinforcement Learning: An Introduction. The notation in that book is slightly different to what you have used, but it explains the concepts involved in your problem is easy-to-understand detail.

What I am trying to understand is, the reward calculation does not take an action as a parameter. How does it choose an action properly.

The reward function does not choose the action. It gives the immediate consequences for your previous action - and in most cases is a consequence of all previous actions, states and random factors in the environment. Although it should only depend on most recent state, previous action plus a random factor at most.

In your case reward depends directly on the resulting state = how many cars at each junction after taking an action, and not directly on the previous action = which traffic flow to enable. It does still depend on the action indirectly, via however the environment works. But that does not need to be expressed mathematically (as a model of the system) for Q learning to work. Although not stated, the reward probably also depends on a random factor outside of your control = how many cars turn up at each junction.

It is the Q value which eventually ranks the different actions and allows you to select the best action. The value $Q^t(s, a)$ gives you the current best estimate for future rewards (for a continuous problem, such as yours, it is common to have a discount factor, $\gamma$, to give more weight to immediate rewards).

There are multiple ways to select the next action in reinforcement learning, depending on the problem, your learning algorithm etc. In the example pseudo-code you copied, the selection process is using Upper Confidence Bound (or UCB), which is a relatively sophisticated way of balancing exploration of actions that might be better than your current best estimate, versus simply using the "best" action so far. If you always took the "best" action, in many cases this would cause a problem because you would never update your knowledge of what the other actions do.

One of the standard textbooks for all this is Sutton & Barto, Reinforcement Learning: An Introduction. The notation in that book is slightly different to what you have used, but it explains the concepts involved in your problem in easy-to-understand detail.

What I am trying to understand is, the reward calculation does not take an action as a parameter. How does it choose an action properly.

The reward function does not choose the action. It gives the immediate consequences for your previous action - and in most cases is a consequence of all previous actions, states and random factors in the environment. Although it should only depend on most recent state, action plus any random factor - in your case it depends on the resulting state = how many cars at each junction, and not on the previous action = which traffic flow to enable. Although not stated, it probably depends on a random factor outside of your control = how many cars turn up at each junction.

It is the Q value which eventually ranks the different actions and allows you to select the best action. The value $Q^t(s, a)$ gives you the current best estimate for future rewards (for a continuous problem, such as yours, it is common to have a discount factor, $\gamma$, to give more weight to immediate rewards).

There are multiple ways to select the next action in reinforcement learning, depending on the problem, your learning algorithm etc. In the example pseudo-code you copied, the selection process is using Upper Confidence Bound (or UCB), which is a relatively sophisticated way of balancing exploration of actions that might be better than your current best estimate, versus simply using the "best" action so far. If you always took the "best" action, in many cases this would cause a problem because you would never update your knowledge of what the other actions do.

The standard textbook for all this is Sutton & Barto, Reinforcement Learning: An Introduction. The notation in that book is slightly different to what you have used, but it explains the concepts involved in your problem is easy-to-understand detail.

What I am trying to understand is, the reward calculation does not take an action as a parameter. How does it choose an action properly.

The reward function does not choose the action. It gives the immediate consequences for your previous action - and in most cases is a consequence of all previous actions, states and random factors in the environment. Although it should only depend on most recent state, previous action plus a random factor at most.

In your case reward depends directly on the resulting state = how many cars at each junction after taking an action, and not directly on the previous action = which traffic flow to enable. It does still depend on the action indirectly, via however the environment works. But that does not need to be expressed mathematically (as a model of the system) for Q learning to work. Although not stated, the reward probably also depends on a random factor outside of your control = how many cars turn up at each junction.

It is the Q value which eventually ranks the different actions and allows you to select the best action. The value $Q^t(s, a)$ gives you the current best estimate for future rewards (for a continuous problem, such as yours, it is common to have a discount factor, $\gamma$, to give more weight to immediate rewards).

There are multiple ways to select the next action in reinforcement learning, depending on the problem, your learning algorithm etc. In the example pseudo-code you copied, the selection process is using Upper Confidence Bound (or UCB), which is a relatively sophisticated way of balancing exploration of actions that might be better than your current best estimate, versus simply using the "best" action so far. If you always took the "best" action, in many cases this would cause a problem because you would never update your knowledge of what the other actions do.

The standard textbook for all this is Sutton & Barto, Reinforcement Learning: An Introduction. The notation in that book is slightly different to what you have used, but it explains the concepts involved in your problem is easy-to-understand detail.

What I am trying to understand is, the reward calculation does not take an action as a parameter. How does it choose an action properly.

The reward function does not choose the action. It gives the immediate consequences for your previous action - and in most cases is a consequence of all previous actions, states and random factors in the environment. Although it should only depend on most recent state, action plus any random factor - in your case it depends on the resulting state = how many cars at each junction, and not on the previous action = which traffic flow to enable. Although not stated, it probably depends on a random factor outside of your control = how many cars turn up at each junction.

It is the Q value which eventually ranks the different actions and allows you to select the best action. The value $Q^t(s, a)$ gives you the current best estimate for future rewards (for a continuous problem, such as yours, it is common to have a discount factor, $\gamma$, to give more weight to immediate rewards).

There are multiple ways to select the next action in reinforcement learning, depending on the problem, your learning algorithm etc. In the example pseudo-code you copied, the selection process is using Upper Confidence Bound (or UCB), which is a relatively sophisticated way of balancing exploration of actions that might be better than your current best estimate, versus simply using the "best" action so far. If you always took the "best" action, in many cases this would cause a problem because you would never update your knowledge of what the other actions do.

The standard textbook for all this is Sutton & Barto, Reinforcement Learning: An Introduction. The notation in that book is slightly different to what you have used, but it explains the concepts involved in your problem is easy-to-understand detail.

What I am trying to understand is, the reward calculation does not take an action as a parameter. How does it choose an action properly.

The reward function does not choose the action. It gives the immediate consequences for your previous action - and in most cases is a consequence of all previous actions, states and random factors in the environment. Although it should only depend on most recent state, action plus any random factor - in your case it depends on the resulting state = how many cars at each junction, and not on the previous action = which traffic flow to enable. Although not stated, it probably depends on a random factor outside of your control = how many cars turn up at each junction.

It is the Q value which eventually ranks the different actions and allows you to select the best action. The value $Q^t(s, a)$ gives you the current best estimate for future rewards (for a continuous problem, such as yours, it is common to have a discount factor, $\gamma$, to give more weight to immediate rewards).

There are multiple ways to select the next action in reinforcement learning, depending on the problem, your learning algorithm etc. In the example pseudo-code you copied, the selection process is using Upper Confidence Bound (or UCB), which is a relatively sophisticated way of balancing exploration of actions that might be better than your current best estimate, versus simply using the "best" action so far. If you always took the "best" action, in many cases this would cause a problem because you would never update your knowledge of what the other actions do.

The standard textbook for all this is Sutton & Barto, Reinforcement Learning: An Introduction. The notation in that book is slightly different to what you have used, but it explains the concepts involved in your problem is easy-to-understand detail.