0
$\begingroup$

I am new to Reinforcement Learning, and I am having trouble understanding the difference between Value function and Q-value

Here is my current understanding: Q-value is just a value for a particular action taken, and the Value function of a state sums over the Q-value of every action possible in that state.

Would this description be accurate?

Thanks!

$\endgroup$
0
$\begingroup$

Here is my current understanding: Q-value is just a value for a particular action taken, and the Value function of a state sums over the Q-value of every action possible in that state.

Would this description be accurate?

Not quite.

First the Q value is a type of value function, it is often called the action value. Both $Q(s,a)$ and the state value function $V(s)$ calculate the expected future return given their parameters, a known environment and a known policy $\pi$ that describes how an agent will select actions in that environment.

In the case of action value $Q(s,a)$, the expected future return is based on the agent taking the action $a$ in state $s$ first, and afterwards following $\pi$. Whilst the state value $V(s)$, the expected future return will depend on what action $\pi$ chooses in state $s$.

There a few ways to write the relationship between $V(s)$ and $Q(s,a)$. If the policy function is deterministic, action in state $s$ given by $\pi(s)$ then:

$$V(s) = Q(s, \pi(s))$$

If the policy function is stochastic, with probability of selecting action $a$ in state $s$ given by $\pi(a|s)$:

$$V(s) = \sum_a \pi(a|s)Q(s, a)$$

This is a sum, as you suggest, but weighted by the probability of taking each action.

$\endgroup$
  • $\begingroup$ Thank you so much for answering, Neil. I am still a little confused...It still kinda sounds like V(s) is the expected future return (which contains value from all possible actions at that state) and Q(s,a) is the expected future return based on ONE specific action taken...Then isn't Q(s,a) kind of like part of V(s)? $\endgroup$ – YCCCCC Sep 25 at 16:17
  • $\begingroup$ @YCCCCC The functions $V(s)$ and $Q(s,a)$ are different but related. Yes one relation is shown in the answer where Q can be viewed as "part of" V. But perhaps a better view of them is that they measure similar things but at different stages of the MDP process. You can use $V(s)$ when all you know is the state - e.g. before you decide what action you might take. You can use $Q(s,a)$ to evaluate a specific action. Which you want to do will depend on other details, such as whether you are choosing actions based on something other than the action value - some algorithms use Q and some use V . . . $\endgroup$ – Neil Slater Sep 25 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.