57 votes

What is the Q function and what is the V function in reinforcement learning?

$V^\pi(s)$ is the "state" value function of an MDP (Markov Decision Process). It's the expected return starting from state $s$ following policy $\pi$: $$V^\pi(s) = E_{\pi} \{G_t \vert s_t = ...
aerin's user avatar
  • 897
55 votes
Accepted

What is "experience replay" and what are its benefits?

The key part of the quoted text is: To perform experience replay we store the agent's experiences $e_t = (s_t,a_t,r_t,s_{t+1})$ This means instead of running Q-learning on state/action pairs as ...
Neil Slater's user avatar
  • 28.8k
47 votes
Accepted

What exactly is bootstrapping in reinforcement learning?

Bootstrapping in RL can be read as "using one or more estimated values in the update step for the same kind of estimated value". In most TD update rules, you will see something like this SARSA(0) ...
Neil Slater's user avatar
  • 28.8k
36 votes
Accepted

What is the Q function and what is the V function in reinforcement learning?

Q-values are a great way to the make actions explicit so you can deal with problems where the transition function is not available (model-free). However, when your action-space is large, things are ...
Juan Leni's user avatar
  • 1,059
19 votes

What is the Q function and what is the V function in reinforcement learning?

You have it right, the $V$ function gives you the value of a state, and $Q$ gives you the value of an action in a state (following a given policy $\pi$). I found the clearest explanation of Q-learning ...
Motorhead's user avatar
  • 291
19 votes
Accepted

Can Reinforcement learning be applied for time series forecasting?

Yes, but in general it is not a good tool for the task, unless there is significant feedback between predictions and ongoing behaviour of the system. To construct a reinforcement learning (RL) ...
Neil Slater's user avatar
  • 28.8k
17 votes

Why do we normalize the discounted rewards when doing policy gradient reinforcement learning?

In general we prefer to normalize the returns for stability purposes. If you work out the backpropagation equations you will see that the return affects the gradients. Thus, we would like to keep its ...
Constantinos's user avatar
  • 2,081
17 votes
Accepted

Prioritized Replay, what does Importance Sampling really do?

DQN suffers intrinsically from instability. In the original implementation, multiple techniques are employed to improve stability: a target network is used with parameters that lag behind the ...
benbotto's user avatar
  • 286
17 votes
Accepted

RL Advantage function why A = Q-V instead of A=V-Q?

In my understanding, $V(s)$ is always larger than $Q(s,a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$ This is incorrect. There is not really such a thing as "...
Neil Slater's user avatar
  • 28.8k
16 votes
Accepted

What is the difference between active learning and reinforcement learning?

Active learning is a technique that is applied to Supervised Learning settings. In the supervised learning paradigm, you train a system by providing inputs and expected outputs (labels). The system ...
noe's user avatar
  • 25.7k
14 votes
Accepted

Difference between AlphaGo's policy network and value network

In brief each net has a different purpose as you mentioned: The value network was used at the leaf nodes to reduce the depth of the tree search. The policy network was used to reduce the breadth of ...
Constantinos's user avatar
  • 2,081
14 votes
Accepted

Q-Learning: Target Network vs Double DQN

Ok, it's simple! Just the "Target network approach": Select an item from Memory Bank Using Target Network, from $S_{t+1}$ determine the index of the best action $A_{t+1}$ and its Q-value Do ...
Kari's user avatar
  • 2,726
13 votes
Accepted

Supervised learning vs reinforcement learning for a simple self driving rc car

I'd suggest you to try a hybrid approach: First, train your car in supervised fashion by demonstration. Just control it and use your commands as labels. This will let you get all the pros of SL. Then,...
rcpinto's user avatar
  • 368
13 votes
Accepted

How does generalised advantage estimation work?

I found very intuitive the explanation of the GAE in the Supplementary material of this paper: DeepMimic. You do not need to read the paper. Just go straight to the Supplementary material section on ...
Constantinos's user avatar
  • 2,081
11 votes
Accepted

Books on Reinforcement Learning

Here you have some good references on Reinforcement Learning: Classic Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, Mass: A Bradford Book; 1998. 322 p. The draft for the ...
Juan Leni's user avatar
  • 1,059
11 votes

Understanding advantage functions

1. Why are the q-values of different actions very close to each other for a given state ? I'm going to explain this with a small example. Consider the game of "Catch". Fruits(circular) keep falling ...
Srinivas K's user avatar
11 votes
Accepted

Reinforcement learning: decreasing loss without increasing reward

How should I interpret this? If a lower loss means more accurate predictions of value, naively I would have expected the agent to take more high-reward actions. A lower loss means more accurate ...
Neil Slater's user avatar
  • 28.8k
10 votes

Difference between AlphaGo's policy network and value network

Here is my concise thought process in understanding the two different networks. First of all, the goal is to find an optimal solution (or very near-optimal) without using an exhaustive search, which ...
Wind P.'s user avatar
  • 101
10 votes
Accepted

AlphaGo (and other game programs using reinforcement-learning) without human database

I'm no expert but it looks like AlphaGo Zero answers your question. https://deepmind.com/blog/alphago-zero-learning-scratch/ Previous versions of AlphaGo initially trained on thousands of human ...
Gabe's user avatar
  • 216
10 votes
Accepted

Why are policy gradient methods preferred over value function approximation in continuous action domains?

But in policy iteration also we are have to output a softmax vector related to each actions This is not strictly true. A softmax vector is one possible way to represent a policy, and works for ...
Neil Slater's user avatar
  • 28.8k
10 votes
Accepted

Why could my DDQN get significantly worse after beating the game repeatedly?

This is called "catastrophic forgetting" and can be a serious problem in many RL scenarios. If you trained a neural network to recognise cats and dogs and did the following: Train it for many ...
Neil Slater's user avatar
  • 28.8k
9 votes

AlphaGo (and other game programs using reinforcement-learning) without human database

The same question has been asked to the author of the AlphaGo paper and his answer was that we don't know what would happen if AlphaGo would learn from scratch (they haven't tested it). However, ...
Constantinos's user avatar
  • 2,081
9 votes

What is a policy in machine learning?

A policy is a state-action mapping. A 'state' is a formalism used in AI that represents the state of the world, i.e. what the agent's idea of the world is. The action is, naturally, what action it ...
tom's user avatar
  • 2,238
9 votes

Can Reinforcement Learning learn to be deceptive?

There is definitely a lot of work to do on the NLP and knowledgebase side of things before you can realise your agent. However, as the question suggests, we can ignore those details and focus on: Can ...
Neil Slater's user avatar
  • 28.8k
8 votes

AlphaGo (and other game programs using reinforcement-learning) without human database

As far as I understood the algorithm of AlphaGo, it is based on a simple reinforcement learning (RL) framework, using Monte-Carlo tree search to select the best actions. On the top of it, the states ...
Robin's user avatar
  • 1,337
8 votes

Difference between AlphaGo's policy network and value network

I think the OP was confusing about AlphaGo with alpha-beta. In alpha-beta, you'd indeed use the policy network for helping with pruning, but not here. Again, there is no pruning as the algorithm ...
SmallChess's user avatar
  • 3,520
8 votes
Accepted

What knowledge do I need in order to write a simple AI program to play a game?

There are multiple ways to approach solving game playing problems. Some games can be solved by search algorithms for example. This works well for card and board games up to some level of complexity. ...
Neil Slater's user avatar
  • 28.8k
8 votes

What is "experience replay" and what are its benefits?

The algorithm (or at least a version of it, as implemented in the Coursera RL capstone project) is as follows: Create a Replay "Buffer" that stores the last ...
Maverick Meerkat's user avatar
8 votes
Accepted

What is "Policy Collapse" and what are the causes?

A web search for "policy collapse" "reinforcement learning" finds this question, a related one in stats.stackexchange.com and the comments section where you found ...
Neil Slater's user avatar
  • 28.8k

Only top scored, non community-wiki answers of a minimum length are eligible