51 votes
Accepted

What is "experience replay" and what are its benefits?

The key part of the quoted text is: To perform experience replay we store the agent's experiences $e_t = (s_t,a_t,r_t,s_{t+1})$ This means instead of running Q-learning on state/action pairs as ...
user avatar
  • 27.2k
48 votes

What is the Q function and what is the V function in reinforcement learning?

$V^\pi(s)$ is the state-value function of an MDP (Markov Decision Process). It's the expected return starting from state $s$ following policy $\pi$. In the expression $$V^\pi(s) = E_{\pi} \{G_t \vert ...
user avatar
  • 757
36 votes
Accepted

What exactly is bootstrapping in reinforcement learning?

Bootstrapping in RL can be read as "using one or more estimated values in the update step for the same kind of estimated value". In most TD update rules, you will see something like this SARSA(0) ...
user avatar
  • 27.2k
29 votes
Accepted

What is the Q function and what is the V function in reinforcement learning?

Q-values are a great way to the make actions explicit so you can deal with problems where the transition function is not available (model-free). However, when your action-space is large, things are ...
user avatar
  • 959
17 votes

Why do we normalize the discounted rewards when doing policy gradient reinforcement learning?

In general we prefer to normalize the returns for stability purposes. If you work out the backpropagation equations you will see that the return affects the gradients. Thus, we would like to keep its ...
user avatar
  • 1,911
17 votes
Accepted

Prioritized Replay, what does Importance Sampling really do?

DQN suffers intrinsically from instability. In the original implementation, multiple techniques are employed to improve stability: a target network is used with parameters that lag behind the ...
user avatar
  • 286
17 votes
Accepted

Can Reinforcement learning be applied for time series forecasting?

Yes, but in general it is not a good tool for the task, unless there is significant feedback between predictions and ongoing behaviour of the system. To construct a reinforcement learning (RL) ...
user avatar
  • 27.2k
15 votes

What is the Q function and what is the V function in reinforcement learning?

You have it right, the $V$ function gives you the value of a state, and $Q$ gives you the value of an action in a state (following a given policy $\pi$). I found the clearest explanation of Q-learning ...
user avatar
  • 251
13 votes
Accepted

RL Advantage function why A = Q-V instead of A=V-Q?

In my understanding, $V(s)$ is always larger than $Q(s,a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$ This is incorrect. There is not really such a thing as "...
user avatar
  • 27.2k
12 votes
Accepted

Difference between AlphaGo's policy network and value network

In brief each net has a different purpose as you mentioned: The value network was used at the leaf nodes to reduce the depth of the tree search. The policy network was used to reduce the breadth of ...
user avatar
  • 1,911
12 votes
Accepted

Supervised learning vs reinforcement learning for a simple self driving rc car

I'd suggest you to try a hybrid approach: First, train your car in supervised fashion by demonstration. Just control it and use your commands as labels. This will let you get all the pros of SL. Then,...
user avatar
  • 358
11 votes
Accepted

Books on Reinforcement Learning

Here you have some good references on Reinforcement Learning: Classic Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, Mass: A Bradford Book; 1998. 322 p. The draft for the ...
user avatar
  • 959
11 votes

Understanding advantage functions

1. Why are the q-values of different actions very close to each other for a given state ? I'm going to explain this with a small example. Consider the game of "Catch". Fruits(circular) keep falling ...
user avatar
11 votes
Accepted

Q-Learning: Target Network vs Double DQN

Ok, it's simple! Just the "Target network approach": Select an item from Memory Bank Using Target Network, from $S_{t+1}$ determine the index of the best action $A_{t+1}$ and its Q-value Do ...
user avatar
  • 2,546
11 votes
Accepted

What is the difference between active learning and reinforcement learning?

Active learning is a technique that is applied to Supervised Learning settings. In the supervised learning paradigm, you train a system by providing inputs and expected outputs (labels). The system ...
user avatar
  • 15k
10 votes

Difference between AlphaGo's policy network and value network

Here is my concise thought process in understanding the two different networks. First of all, the goal is to find an optimal solution (or very near-optimal) without using an exhaustive search, which ...
user avatar
  • 101
10 votes
Accepted

AlphaGo (and other game programs using reinforcement-learning) without human database

I'm no expert but it looks like AlphaGo Zero answers your question. https://deepmind.com/blog/alphago-zero-learning-scratch/ Previous versions of AlphaGo initially trained on thousands of human ...
user avatar
  • 216
10 votes
Accepted

How does generalised advantage estimation work?

I found very intuitive the explanation of the GAE in the Supplementary material of this paper: DeepMimic. You do not need to read the paper. Just go straight to the Supplementary material section on ...
user avatar
  • 1,911
9 votes
Accepted

Does reinforcement learning require the help of other learning algorithms?

You do not need additional learning algorithms to perform reinforcement learning in simple systems where you can explore all states. For those, simple iterative Q-learning can do very well - as well ...
user avatar
  • 27.2k
9 votes

AlphaGo (and other game programs using reinforcement-learning) without human database

The same question has been asked to the author of the AlphaGo paper and his answer was that we don't know what would happen if AlphaGo would learn from scratch (they haven't tested it). However, ...
user avatar
  • 1,911
9 votes

Can Reinforcement Learning learn to be deceptive?

There is definitely a lot of work to do on the NLP and knowledgebase side of things before you can realise your agent. However, as the question suggests, we can ignore those details and focus on: Can ...
user avatar
  • 27.2k
8 votes

AlphaGo (and other game programs using reinforcement-learning) without human database

As far as I understood the algorithm of AlphaGo, it is based on a simple reinforcement learning (RL) framework, using Monte-Carlo tree search to select the best actions. On the top of it, the states ...
user avatar
  • 1,277
8 votes
Accepted

What knowledge do I need in order to write a simple AI program to play a game?

There are multiple ways to approach solving game playing problems. Some games can be solved by search algorithms for example. This works well for card and board games up to some level of complexity. ...
user avatar
  • 27.2k
8 votes
Accepted

Why are policy gradient methods preferred over value function approximation in continuous action domains?

But in policy iteration also we are have to output a softmax vector related to each actions This is not strictly true. A softmax vector is one possible way to represent a policy, and works for ...
user avatar
  • 27.2k
8 votes
Accepted

Policy-based RL method - how do continuous actions look like?

The main requirement of on-policy policy gradient methods is that they use a parametric policy $\pi(a|s, \theta)$ that is differentiable with respect to the parameters $\theta$. This is not ...
user avatar
  • 27.2k
8 votes
Accepted

Reinforcement learning: decreasing loss without increasing reward

How should I interpret this? If a lower loss means more accurate predictions of value, naively I would have expected the agent to take more high-reward actions. A lower loss means more accurate ...
user avatar
  • 27.2k
8 votes
Accepted

Why could my DDQN get significantly worse after beating the game repeatedly?

This is called "catastrophic forgetting" and can be a serious problem in many RL scenarios. If you trained a neural network to recognise cats and dogs and did the following: Train it for many ...
user avatar
  • 27.2k
7 votes

Difference between AlphaGo's policy network and value network

I think the OP was confusing about AlphaGo with alpha-beta. In alpha-beta, you'd indeed use the policy network for helping with pruning, but not here. Again, there is no pruning as the algorithm ...
user avatar
  • 3,348
7 votes

Books on Reinforcement Learning

There is a free online course on Reinforcement Learning by Udacity. Check : Machine Learning: Reinforcement Learning
user avatar
  • 1,860
7 votes

What knowledge do I need in order to write a simple AI program to play a game?

It highly depends on the type of game and the information about the state of the game that is available to your AI. Some of the most famous game playing AIs from last few years are based on deep ...
user avatar
  • 15k

Only top scored, non community-wiki answers of a minimum length are eligible