# Tag Info

Accepted

### What is "experience replay" and what are its benefits?

The key part of the quoted text is: To perform experience replay we store the agent's experiences $e_t = (s_t,a_t,r_t,s_{t+1})$ This means instead of running Q-learning on state/action pairs as ...
• 27.2k

### What is the Q function and what is the V function in reinforcement learning?

$V^\pi(s)$ is the state-value function of an MDP (Markov Decision Process). It's the expected return starting from state $s$ following policy $\pi$. In the expression V^\pi(s) = E_{\pi} \{G_t \vert ...
• 757
Accepted

### What exactly is bootstrapping in reinforcement learning?

Bootstrapping in RL can be read as "using one or more estimated values in the update step for the same kind of estimated value". In most TD update rules, you will see something like this SARSA(0) ...
• 27.2k
Accepted

### What is the Q function and what is the V function in reinforcement learning?

Q-values are a great way to the make actions explicit so you can deal with problems where the transition function is not available (model-free). However, when your action-space is large, things are ...
• 959

### Why do we normalize the discounted rewards when doing policy gradient reinforcement learning?

In general we prefer to normalize the returns for stability purposes. If you work out the backpropagation equations you will see that the return affects the gradients. Thus, we would like to keep its ...
• 1,911
Accepted

### Prioritized Replay, what does Importance Sampling really do?

DQN suffers intrinsically from instability. In the original implementation, multiple techniques are employed to improve stability: a target network is used with parameters that lag behind the ...
• 286
Accepted

### Can Reinforcement learning be applied for time series forecasting?

Yes, but in general it is not a good tool for the task, unless there is significant feedback between predictions and ongoing behaviour of the system. To construct a reinforcement learning (RL) ...
• 27.2k

### What is the Q function and what is the V function in reinforcement learning?

You have it right, the $V$ function gives you the value of a state, and $Q$ gives you the value of an action in a state (following a given policy $\pi$). I found the clearest explanation of Q-learning ...
• 251
Accepted

In my understanding, $V(s)$ is always larger than $Q(s,a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$ This is incorrect. There is not really such a thing as "...
• 27.2k
Accepted

### Difference between AlphaGo's policy network and value network

In brief each net has a different purpose as you mentioned: The value network was used at the leaf nodes to reduce the depth of the tree search. The policy network was used to reduce the breadth of ...
• 1,911
Accepted

### Supervised learning vs reinforcement learning for a simple self driving rc car

I'd suggest you to try a hybrid approach: First, train your car in supervised fashion by demonstration. Just control it and use your commands as labels. This will let you get all the pros of SL. Then,...
• 358
Accepted

### Books on Reinforcement Learning

Here you have some good references on Reinforcement Learning: Classic Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, Mass: A Bradford Book; 1998. 322 p. The draft for the ...
• 959

1. Why are the q-values of different actions very close to each other for a given state ? I'm going to explain this with a small example. Consider the game of "Catch". Fruits(circular) keep falling ...
• 211
Accepted

### Q-Learning: Target Network vs Double DQN

Ok, it's simple! Just the "Target network approach": Select an item from Memory Bank Using Target Network, from $S_{t+1}$ determine the index of the best action $A_{t+1}$ and its Q-value Do ...
• 2,546
Accepted

### What is the difference between active learning and reinforcement learning?

Active learning is a technique that is applied to Supervised Learning settings. In the supervised learning paradigm, you train a system by providing inputs and expected outputs (labels). The system ...
• 15k

### Difference between AlphaGo's policy network and value network

Here is my concise thought process in understanding the two different networks. First of all, the goal is to find an optimal solution (or very near-optimal) without using an exhaustive search, which ...
• 101
Accepted

### AlphaGo (and other game programs using reinforcement-learning) without human database

I'm no expert but it looks like AlphaGo Zero answers your question. https://deepmind.com/blog/alphago-zero-learning-scratch/ Previous versions of AlphaGo initially trained on thousands of human ...
• 216
Accepted

### How does generalised advantage estimation work?

I found very intuitive the explanation of the GAE in the Supplementary material of this paper: DeepMimic. You do not need to read the paper. Just go straight to the Supplementary material section on ...
• 1,911
Accepted

### Does reinforcement learning require the help of other learning algorithms?

You do not need additional learning algorithms to perform reinforcement learning in simple systems where you can explore all states. For those, simple iterative Q-learning can do very well - as well ...
• 27.2k

### AlphaGo (and other game programs using reinforcement-learning) without human database

The same question has been asked to the author of the AlphaGo paper and his answer was that we don't know what would happen if AlphaGo would learn from scratch (they haven't tested it). However, ...
• 1,911

### Can Reinforcement Learning learn to be deceptive?

There is definitely a lot of work to do on the NLP and knowledgebase side of things before you can realise your agent. However, as the question suggests, we can ignore those details and focus on: Can ...
• 27.2k

### AlphaGo (and other game programs using reinforcement-learning) without human database

As far as I understood the algorithm of AlphaGo, it is based on a simple reinforcement learning (RL) framework, using Monte-Carlo tree search to select the best actions. On the top of it, the states ...
• 1,277
Accepted

### What knowledge do I need in order to write a simple AI program to play a game?

There are multiple ways to approach solving game playing problems. Some games can be solved by search algorithms for example. This works well for card and board games up to some level of complexity. ...
• 27.2k
Accepted

### Why are policy gradient methods preferred over value function approximation in continuous action domains?

But in policy iteration also we are have to output a softmax vector related to each actions This is not strictly true. A softmax vector is one possible way to represent a policy, and works for ...
• 27.2k
Accepted

### Policy-based RL method - how do continuous actions look like?

The main requirement of on-policy policy gradient methods is that they use a parametric policy $\pi(a|s, \theta)$ that is differentiable with respect to the parameters $\theta$. This is not ...
• 27.2k
Accepted

### Reinforcement learning: decreasing loss without increasing reward

How should I interpret this? If a lower loss means more accurate predictions of value, naively I would have expected the agent to take more high-reward actions. A lower loss means more accurate ...
• 27.2k
Accepted

### Why could my DDQN get significantly worse after beating the game repeatedly?

This is called "catastrophic forgetting" and can be a serious problem in many RL scenarios. If you trained a neural network to recognise cats and dogs and did the following: Train it for many ...
• 27.2k

### Difference between AlphaGo's policy network and value network

I think the OP was confusing about AlphaGo with alpha-beta. In alpha-beta, you'd indeed use the policy network for helping with pruning, but not here. Again, there is no pruning as the algorithm ...
• 3,348