Timeline for In reinforcement learning, why learn Q rather than V?
Current License: CC BY-SA 4.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 25 at 21:27 | vote | accept | xyzzyrz | ||
| Mar 25 at 21:27 | comment | added | xyzzyrz | I think I see the problem. In RL, the environment rewards based on actions (conditioned on state), and not simply arriving in a state. So my initial assumption that Q(s,a)=V(s') is incorrect. In other words, two different states that transition to the same next state can have different rewards, despite arriving at the same next state. Thank you. | |
| Mar 24 at 20:14 | comment | added | J_H | You reduced the 1m range (thousand positions) to 1cm (ten positions); that's cool. I had contemplated a "single move universe" for simplicity, where failing to thread is a fatal error, similar to a single life Pitfall Harry falling down a hole. Maybe I should have added that the atomic action is {move left X, move up 1cm, gripper releases thread, return to origin by moving right X}. Now it's starting to resemble a carnival game or a single-bullet FPS. We agree that Q and V each have their uses. | |
| Mar 24 at 20:03 | comment | added | xyzzyrz | Thanks for the example. However the state space here is not just the initial condition, right? Let's say there are 10 distinct millimeters that you can move left before attempting to thread. This means there are up to 11 next (terminal) states. So in this world, you can simply evaluate the reward from each of those states instead of each action emerging from the initial state. | |
| Mar 24 at 16:57 | history | edited | J_H | CC BY-SA 4.0 |
added 34 characters in body
|
| Mar 22 at 19:39 | history | edited | J_H | CC BY-SA 4.0 |
added 2 characters in body
|
| Mar 22 at 19:21 | history | edited | J_H | CC BY-SA 4.0 |
add a tailor example
|
| Mar 22 at 19:17 | comment | added | J_H | It is total because there's different assumptions: "... with a value defined for any thoroughly explored state" In contrast š¯‘„ is of interest while we're still exploring, and it makes explicit our very partial knowledge (so far) of the world. I confess there may be better terms for this situation than {"partial", "total"}, and would be happy to accept proposals. | |
| Mar 22 at 19:14 | history | edited | J_H | CC BY-SA 4.0 |
add a tailor example
|
| Mar 22 at 18:30 | comment | added | xyzzyrz | Why is v total while q partial? In q learning, for example, these are both just tables of values that we iteratively update, so I think I'm not seeing the profound distinction. As for telling the mdp to avoid states known to have low values for some actions, that is inherent in the tentative v(s') value being low, and doesn't require q. Perhaps this would be most productive: are you able to convey your point using a simple example with respect to, say, grid world? | |
| Mar 21 at 17:43 | history | answered | J_H | CC BY-SA 4.0 |