Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Filter by
Sorted by
Tagged with
20
votes
2answers
29k views

What is “experience replay” and what are its benefits?

I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
8
votes
1answer
656 views

Reward dependent on (state, action) versus (state, action, successor state)

I am studying reinforcement learning and I am working methodically through Sutton and Barto's book plus David Silver's lectures. I have noticed a minor difference in how the Markov Decision Processes ...
8
votes
2answers
2k views

Prioritized Replay, what does Importance Sampling really do?

I can't understand the purpose of importance-sampling weights (IS) in Prioritized Replay (page 5). A transition is more likely to be sampled from experience replay the larger its "cost" is. My ...
3
votes
1answer
3k views

RL Advantage function why A = Q-V instead of A=V-Q?

In RL Course by David Silver - Lecture 7: Policy Gradient Methods, David explains what an Advantage function is, and how it's the difference between Q(s,a) and the V(s) Preliminary, from this post: ...
4
votes
2answers
649 views

Hindsight Experience Replay, how to define a partially-known End-Goal

One of the requirements of the Hindsight Experience Replay is supplying the DQN with a state and a goal (the desired end-state) that we hope to end up in: ...
32
votes
5answers
24k views

What is the Q function and what is the V function in reinforcement learning?

It seems to me that the $V$ function can be easily expressed by the $Q$ function and thus the $V$ function seems to be superfluous to me. However, I'm new to reinforcement learning so I guess I got ...
24
votes
2answers
7k views

What exactly is bootstrapping in reinforcement learning?

Apparently, in reinforcement learning, temporal-difference (TD) method is a bootstrapping method. On the other hand, Monte Carlo methods are not bootstrapping methods. What exactly is bootstrapping ...
12
votes
1answer
4k views

Supervised learning vs reinforcement learning for a simple self driving rc car

I'm building a remote-controlled self driving car for fun. I'm using a Raspberry Pi as the onboard computer; and I'm using various plug-ins, such as a Raspberry Pi camera and distance sensors, for ...
9
votes
4answers
2k views

Books on Reinforcement Learning

I have been trying to understand reinforcement learning for quite sometime, but somehow I am not able to visualize how to write a program for reinforcement learning to solve a grid world problem. Can ...
7
votes
2answers
4k views

Why are policy gradient methods preferred over value function approximation in continuous action domains?

In value-function approximation, in particular, in deep Q-learning, I understand that we first predict the Q values for each action. However, when there are many actions, this task is not easy. But ...
4
votes
1answer
938 views

How does Implicit Quantile-Regression Network (IQN) differ from QR-DQN?

For several months I browsed the internet hoping to find a user-friendly explanation of the Implicit Quantile Regression Network (IQN). But, it seems there is none at all. How does IQN differ from ...
5
votes
1answer
1k views

What is Compatible Function Approximation theorem in reinforcement learning?

I am following David Silver's RL course. In the policy gradient section, I found this slide that I would like have an explanation of. What are these two conditions? What is the logic behind the ...
3
votes
1answer
1k views

Why random sample from replay for DQN?

I'm trying to gain an intuitive understanding of deep reinforcement learning. In deep Q-networks (DQN) we store all actions/environments/rewards in a memory array and at the end of the episode, "...
3
votes
1answer
32 views

Trouble understanding the partial differentiation used in reinforcement learning

I am studying deterministic actor-critic algorithms in reinforcement learning. I try to give a brief explanation of actor-critic algorithms before jumping into the mathematics. The actor takes in ...
14
votes
4answers
456 views

AlphaGo (and other game programs using reinforcement-learning) without human database

I am not a specialist of the subject, and my question is probably very naive. It stems from an essay to understand the powers and limitation of reinforcement learning as used in the AlphaGo program. ...
2
votes
2answers
532 views

Auto-Encoder to condense (pre-process) large one-hot input vectors?

In my 3D game there are 300 categories to which a creature can belong. I would like to teach my RL agent to make decisions based on its 10 closest monsters So far, my Neural Network input vector is ...
2
votes
1answer
98 views

Reinforcement learning: easily learnable state representation

I have created a simple OpenAI Gym environment, which consists of: A continuous 2D world with x and y in range [0.0, 1.0] A rabbit which slowly moves randomly in the world with a constant speed A '...
2
votes
1answer
163 views

What is wrong with this reinforcement learning environment ?

I'm working on below reinforcement learning problem: I have bottle of fix capacity (say 5 liters). At the bottom of bottle there is cock to remove water. The distribution of removal of water is not ...
3
votes
1answer
223 views

Catastrophic forgetting in linear semi-gradient RL agent?

I've been working through the Sutton + Barto RL text, implementing a number of the algos + running them in the OpenAI gym. One phenomenon that I seem to come across quite regularly is that agents who, ...
3
votes
1answer
1k views

Dueling DQN - can't understand its mechanism

I am trying to understand the purpose of Dueling DQN. According to this blogpost: our reinforcement learning agent may not need to care about both value and advantage at any given time - this seems ...
3
votes
1answer
296 views

How is that possible that a reward function depends both on the next state and an action from current state?

There's the concept of "expected value of the next reward", often denoted as $\mathcal{R}^a_{ss'}$, and defined as $$ \mathcal{R}^a_{ss'} = \mathbb{E}\left(r_{t+1} \mid s_t = s, a_t = a, s_{t+1} = s'...
2
votes
1answer
93 views

Dueling Network gradient with respect to Advantage stream

Looking at Dueling DQN: $Q = V + A - mean(A)$ For simplicity, let's assume we are working with 4 neurons. Recall that Value stream only has 1 neuron $(v_0)$ Re-writing the above equation, we get: $...
2
votes
1answer
535 views

What are features in the context of reinforcement learning?

In machine learning, "feature" is a synonym for explanatory variables. I know what a feature is. However, in the specific case of RL, it's not clear to me what features are. What are "features" in the ...
1
vote
1answer
90 views

Cannot see what the “notation abuse” is, mentioned by author of book

From Sutton and Barto, Reinforcement Learning: An Introduction (second edition draft), in equation 3.4 of page 38. The probabilities given by the four-argument function p completely characterize ...
1
vote
0answers
548 views

DQN cannot learn or converge

I have implemented a DQN using keras. The task is to collect the circles and avoid the red circle and crosses. The associated rewards are +5, -5 and 0 otherwise. if the agent go out of the board, the ...
1
vote
1answer
1k views

Convergence of vanilla or natural policy gradients (e.g. REINFORCE)

I am reading in a lot of places that policy gradients, especially vanilla and natural, are at least guaranteed to converge to a local optimum (see, e.g., pg. 2 of Policy Gradient Methods for Robotics ...
0
votes
2answers
44 views

RL Invertible Value Function approach - why it prevents rewards from exploding?

Authors of "Recurrent Experience Replay in Destributed RL", page 3, use the function $h$ to prevent rewards from exploding: $$h(x) = \operatorname{sign}(x)(\sqrt{|x|+1} -1) + \epsilon x$$ where $\...