57
votes
What is the Q function and what is the V function in reinforcement learning?
$V^\pi(s)$ is the "state" value function of an MDP (Markov Decision Process). It's the expected return starting from state $s$ following policy $\pi$:
$$V^\pi(s) = E_{\pi} \{G_t \vert s_t = ...
- 897
55
votes
Accepted
What is "experience replay" and what are its benefits?
The key part of the quoted text is:
To perform experience replay we store the agent's experiences $e_t = (s_t,a_t,r_t,s_{t+1})$
This means instead of running Q-learning on state/action pairs as ...
- 28.6k
47
votes
Accepted
What exactly is bootstrapping in reinforcement learning?
Bootstrapping in RL can be read as "using one or more estimated values in the update step for the same kind of estimated value".
In most TD update rules, you will see something like this SARSA(0) ...
- 28.6k
36
votes
Accepted
What is the Q function and what is the V function in reinforcement learning?
Q-values are a great way to the make actions explicit so you can deal with problems where the transition function is not available (model-free). However, when your action-space is large, things are ...
- 1,049
19
votes
What is the Q function and what is the V function in reinforcement learning?
You have it right, the $V$ function gives you the value of a state, and $Q$ gives you the value of an action in a state (following a given policy $\pi$). I found the clearest explanation of Q-learning ...
- 291
19
votes
Accepted
Can Reinforcement learning be applied for time series forecasting?
Yes, but in general it is not a good tool for the task, unless there is significant feedback between predictions and ongoing behaviour of the system.
To construct a reinforcement learning (RL) ...
- 28.6k
17
votes
Why do we normalize the discounted rewards when doing policy gradient reinforcement learning?
In general we prefer to normalize the returns for stability purposes. If you work out the backpropagation equations you will see that the return affects the gradients. Thus, we would like to keep its ...
- 2,061
17
votes
Accepted
Prioritized Replay, what does Importance Sampling really do?
DQN suffers intrinsically from instability. In the original implementation, multiple techniques are employed to improve stability:
a target network is used with parameters that lag behind the ...
- 286
17
votes
Accepted
RL Advantage function why A = Q-V instead of A=V-Q?
In my understanding, $V(s)$ is always larger than $Q(s,a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$
This is incorrect. There is not really such a thing as "...
- 28.6k
16
votes
Accepted
What is the difference between active learning and reinforcement learning?
Active learning is a technique that is applied to Supervised Learning settings. In the supervised learning paradigm, you train a system by providing inputs and expected outputs (labels). The system ...
- 23.8k
14
votes
Accepted
Difference between AlphaGo's policy network and value network
In brief each net has a different purpose as you mentioned:
The value network was used at the leaf nodes to reduce the depth of the tree search.
The policy network was used to reduce the breadth of ...
- 2,061
14
votes
Accepted
Q-Learning: Target Network vs Double DQN
Ok, it's simple!
Just the "Target network approach":
Select an item from Memory Bank
Using Target Network, from $S_{t+1}$ determine the index of the best action $A_{t+1}$ and its Q-value
Do ...
- 2,706
13
votes
Accepted
Supervised learning vs reinforcement learning for a simple self driving rc car
I'd suggest you to try a hybrid approach:
First, train your car in supervised fashion by demonstration. Just
control it and use your commands as labels. This will let you get all
the pros of SL.
Then,...
- 368
13
votes
Accepted
How does generalised advantage estimation work?
I found very intuitive the explanation of the GAE in the Supplementary material of this paper: DeepMimic. You do not need to read the paper. Just go straight to the Supplementary material section on ...
- 2,061
11
votes
Accepted
Books on Reinforcement Learning
Here you have some good references on Reinforcement Learning:
Classic
Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, Mass: A Bradford Book; 1998. 322 p.
The draft for the ...
- 1,049
11
votes
Understanding advantage functions
1. Why are the q-values of different actions very close to each other for a given state ?
I'm going to explain this with a small example.
Consider the game of "Catch". Fruits(circular) keep falling ...
- 211
10
votes
Accepted
Does reinforcement learning require the help of other learning algorithms?
You do not need additional learning algorithms to perform reinforcement learning in simple systems where you can explore all states. For those, simple iterative Q-learning can do very well - as well ...
- 28.6k
10
votes
Difference between AlphaGo's policy network and value network
Here is my concise thought process in understanding the two different networks.
First of all, the goal is to find an optimal solution (or very near-optimal) without using an exhaustive search, which ...
- 101
10
votes
Accepted
AlphaGo (and other game programs using reinforcement-learning) without human database
I'm no expert but it looks like AlphaGo Zero answers your question.
https://deepmind.com/blog/alphago-zero-learning-scratch/
Previous versions of AlphaGo initially trained on thousands of human
...
- 216
10
votes
Accepted
Why are policy gradient methods preferred over value function approximation in continuous action domains?
But in policy iteration also we are have to output a softmax vector related to each actions
This is not strictly true. A softmax vector is one possible way to represent a policy, and works for ...
- 28.6k
10
votes
Accepted
Reinforcement learning: decreasing loss without increasing reward
How should I interpret this? If a lower loss means more accurate predictions of value, naively I would have expected the agent to take more high-reward actions.
A lower loss means more accurate ...
- 28.6k
10
votes
Accepted
Why could my DDQN get significantly worse after beating the game repeatedly?
This is called "catastrophic forgetting" and can be a serious problem in many RL scenarios.
If you trained a neural network to recognise cats and dogs and did the following:
Train it for many ...
- 28.6k
9
votes
AlphaGo (and other game programs using reinforcement-learning) without human database
The same question has been asked to the author of the AlphaGo paper and his answer was that we don't know what would happen if AlphaGo would learn from scratch (they haven't tested it).
However, ...
- 2,061
9
votes
What is a policy in machine learning?
A policy is a state-action mapping. A 'state' is a formalism used in AI that represents the state of the world, i.e. what the agent's idea of the world is. The action is, naturally, what action it ...
- 2,238
9
votes
Can Reinforcement Learning learn to be deceptive?
There is definitely a lot of work to do on the NLP and knowledgebase side of things before you can realise your agent. However, as the question suggests, we can ignore those details and focus on: Can ...
- 28.6k
8
votes
Difference between AlphaGo's policy network and value network
I think the OP was confusing about AlphaGo with alpha-beta. In alpha-beta, you'd indeed use the policy network for helping with pruning, but not here. Again, there is no pruning as the algorithm ...
- 3,520
8
votes
AlphaGo (and other game programs using reinforcement-learning) without human database
As far as I understood the algorithm of AlphaGo, it is based on a simple reinforcement learning (RL) framework, using Monte-Carlo tree search to select the best actions. On the top of it, the states ...
- 1,307
8
votes
Accepted
What knowledge do I need in order to write a simple AI program to play a game?
There are multiple ways to approach solving game playing problems. Some games can be solved by search algorithms for example. This works well for card and board games up to some level of complexity. ...
- 28.6k
8
votes
What is "experience replay" and what are its benefits?
The algorithm (or at least a version of it, as implemented in the Coursera RL capstone project) is as follows:
Create a Replay "Buffer" that stores the last ...
- 221
8
votes
Accepted
What is "Policy Collapse" and what are the causes?
A web search for "policy collapse" "reinforcement learning" finds this question, a related one in stats.stackexchange.com and the comments section where you found ...
- 28.6k
Only top scored, non community-wiki answers of a minimum length are eligible