# Reinforcement Learning (Q Learning)

I was reading a paper on traffic flow optimization using Multi-Agent Q learning. the paper proposes the following method: Deploy a Reinforcement learning controller at each intersection with traffic lights.

first the Q value equation is:

$Q^{t}(s,a) = (1- \alpha )Q^{t-1}(s,a) + \alpha (R{t} + \gamma max_{a}(Q^{t-1}(s,a))$

second the state is: the sum of vehicle queues lengths at the current intersection and one hop intersections

third the action space is: here the actions represent the possible moves of vehicles at the intersection.

forth the reward at time t is

$R^{t} = -( w_1\sum q_{current intersection} + w_2\sum q_{neighbors} )$

where the q refers to vehicle queues lengths, w1 and w2 are constants.

fifth there is the algorithm in the image below where it acquires the action required for maximizing the Q value What I am trying to understand is, the reward calculation does not take an action as a parameter. How does it choose an action properly.

I am a newbie to reinforcement learning, so please if you find my question naive, cosedire referring me to a proper textbook. Thanks

• This shows little effort in trying to make a proper question. First, as suggested by @NeilSlater, consider inserting the equations properly in the text and also provide background, state what you have understood so far and what exact information do you need in the potential answers. I can probably help you clarifying some concepts if I properly understand what are you trying to ask here. Mar 5 '18 at 14:00
• I am sorry i'm just a newbie here and the way I proposed my question may not be practical. This is a method for action selection I am aware of that. I will edit the question to make it better. Mar 5 '18 at 17:07
• @AdnanSaood Pls cite the paper. Mar 6 '18 at 14:47
• @horaceT Liu, Y., Liu, L., & Chen, W. P. (2017). Intelligent Traffic Light Control Using Distributed Multi-agent Q Learning. arXiv preprint arXiv:1711.10941. Mar 6 '18 at 18:49

It is the Q value which eventually ranks the different actions and allows you to select the best action. The value $Q^t(s, a)$ gives you the current best estimate for future rewards (for a continuous problem, such as yours, it is common to have a discount factor, $\gamma$, to give more weight to immediate rewards).