Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Filter by
Sorted by
Tagged with
32
votes
5answers
23k views

What is the Q function and what is the V function in reinforcement learning?

It seems to me that the $V$ function can be easily expressed by the $Q$ function and thus the $V$ function seems to be superfluous to me. However, I'm new to reinforcement learning so I guess I got ...
25
votes
5answers
13k views

Difference between AlphaGo's policy network and value network

I was reading a high level summary about Google's AlphaGo (http://googleresearch.blogspot.co.uk/2016/01/alphago-mastering-ancient-game-of-go.html), and I came across the terms "policy network" and "...
23
votes
2answers
6k views

What exactly is bootstrapping in reinforcement learning?

Apparently, in reinforcement learning, temporal-difference (TD) method is a bootstrapping method. On the other hand, Monte Carlo methods are not bootstrapping methods. What exactly is bootstrapping ...
18
votes
1answer
28k views

What is “experience replay” and what are its benefits?

I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
14
votes
4answers
452 views

AlphaGo (and other game programs using reinforcement-learning) without human database

I am not a specialist of the subject, and my question is probably very naive. It stems from an essay to understand the powers and limitation of reinforcement learning as used in the AlphaGo program. ...
12
votes
1answer
3k views

Supervised learning vs reinforcement learning for a simple self driving rc car

I'm building a remote-controlled self driving car for fun. I'm using a Raspberry Pi as the onboard computer; and I'm using various plug-ins, such as a Raspberry Pi camera and distance sensors, for ...
10
votes
2answers
455 views

implementing temporal difference in chess

I have been developing a chess program which makes use of alpha-beta pruning algorithm and an evaluation function that evaluates positions using the following features namely material, kingsafety, ...
10
votes
2answers
292 views

Cooperative Reinforcement Learning

I already have a functioning $Q(\lambda)$ implementation for a single agent working on a dynamic pricing problem with the goal of maximizing revenue. The problem that I'm working with, however, ...
9
votes
4answers
2k views

Books on Reinforcement Learning

I have been trying to understand reinforcement learning for quite sometime, but somehow I am not able to visualize how to write a program for reinforcement learning to solve a grid world problem. Can ...
9
votes
1answer
4k views

Can Reinforcement learning be applied for time series forecasting?

Can Reinforcement learning be applied for time series forecasting?
8
votes
3answers
657 views

What knowledge do I need in order to write a simple AI program to play a game?

I'm a B.Sc graduate. One of my courses was 'Introduction to Machine Learning', and I always wanted to do a personal project in this subject. I recently heard about different AI training to play games ...
8
votes
2answers
870 views

Is this a Q-learning algorithm or just brute force?

I have been playing with an algorithm that learns how to play tictactoe. The basic pseudocode is: ...
8
votes
2answers
6k views

Why do we normalize the discounted rewards when doing policy gradient reinforcement learning?

I'm trying to understand the policy gradient approach for solving the cartpole problem. In this approach, we're expressing the gradient of the loss w.r.t each parameter of our policy as an expectation ...
8
votes
2answers
2k views

Prioritized Replay, what does Importance Sampling really do?

I can't understand the purpose of importance-sampling weights (IS) in Prioritized Replay (page 5). A transition is more likely to be sampled from experience replay the larger its "cost" is. My ...
7
votes
2answers
4k views

Why are policy gradient methods preferred over value function approximation in continuous action domains?

In value-function approximation, in particular, in deep Q-learning, I understand that we first predict the Q values for each action. However, when there are many actions, this task is not easy. But ...
7
votes
2answers
2k views

How does generalised advantage estimation work?

I've been trying to add GAE to my A2C implementation for a while now, but I' can't quite seem to grok how it works. My understanding of it, is that it reduces the variance of the advantage estimation ...
7
votes
2answers
787 views

What is the novelty in AlphaGo, Google Deepmind's Go playing system?

Recently researchers at Google DeepMind published a paper, where they described a Go playing system that beat the best current computer programs and the human European champion. I had a quick look at ...
7
votes
2answers
2k views

Why does Q Learning diverge?

My Q-Learning algorithm's state values keep on diverging to infinity, which means my weights are diverging too. I use a neural network for my value-mapping. I've tried: Clipping the "reward + ...
7
votes
2answers
3k views

How to teach neural network a policy for a board game using reinforcement learning?

I need to use reinforcement learning to teach a neural net a policy for a board game. I chose Q-learining as the specific alghoritm. I'd like a neural net to have the following structure: layer - <...
7
votes
0answers
137 views

Formal proof of vanilla policy gradient convergence

So I stumbled upon this question, where the author asks for a proof of vanilla policy gradient procedures. The answer provided points to some literature, but the formal proof is nowhere to be included....
7
votes
2answers
122 views

Representing similar states in reinforcement learning?

Let's say I'd like to design a Q learning algorithm that learns to play poker. The number of different possible States is very large, but a lot are very similar: for example, if the initial state ...
6
votes
4answers
1k views

Does reinforcement learning require the help of other learning algorithms?

Can't reinforcement learning be used without the help of other learning algorithms like SVM and MLP back propagation? I consulted two papers: Paper 1 Paper 2 both have used other machine learning ...
6
votes
5answers
476 views

What is Reinforcement Learning?

I am familiar with the concepts of Supervised and Unsupervised Learning but recently Reinforcement (Reinforced?) Learning also popped up before me a couple of times. Could anyone give a hint what is ...
6
votes
2answers
3k views

RL Policy Gradient: How to deal with rewards that are strictly positive?

In short: In the policy gradient method, if the reward is always positive (never negative), the policy gradient will always be positive, hence it will keep making our parameters larger. This makes ...
6
votes
1answer
5k views

What is the difference between “expected return” and “expected reward” in the context of RL?

The value of a state $s$ under a certain policy $\pi$, $V^\pi(s)$, is defined as the "expected return" starting from state $s$. More precisely, it is defined as $$ V^\pi(s) = \mathbb{E}\left(R_t \mid ...
6
votes
1answer
11k views

Text extraction from documents using NLP or Deep Learning

I am looking for references(Papers/github projects) on how to use deep learning in a text extraction task. Recently I was given a task to extract important information from documents of similar type, ...
6
votes
2answers
73 views

Actor-critic architecture: How is the policy updated?

I am going through the ddpg baseline code to try and gain an intuitive understanding of how the actor and critic networks function. DDPG has two components: the actor which is the deterministic ...
6
votes
1answer
623 views

Reward dependent on (state, action) versus (state, action, successor state)

I am studying reinforcement learning and I am working methodically through Sutton and Barto's book plus David Silver's lectures. I have noticed a minor difference in how the Markov Decision Processes ...
5
votes
1answer
6k views

Understanding advantage functions

The paper explaining 'Advantage Updating' as a method to improve Q-learning uses the following as its motivation. Q-learning requires relatively little computation per update, but it is useful to ...
5
votes
2answers
2k views

What is a policy in machine learning?

While I was reading the paper "Grounded Action Transformation for Robot Learning in Simulation", I came across the term "policy". Could someone explain to me what that actually is (in general and in ...
5
votes
1answer
2k views

Reinforcement learning: decreasing loss without increasing reward

I'm trying to solve OpenAI Gym's LunarLander-v2. I'm using the Deep Q-Learning algorithm. I have tried various hyperparameters, but I can't get a good score. Generally the loss decreases over many ...
5
votes
2answers
100 views

Why a Random Reward in One-step Dynamics MDP?

I am reading the 2018 book by Sutton & Barto on Reinforcement Learning and I am wondering the benefit of defining the one-step dynamics of an MDP as $$ p(s',r|s,a) = Pr(S_{t+1},R_{t+1}|S_t=s, ...
5
votes
1answer
4k views

Q-Learning: Target Network vs Double DQN

I am having a hard time understanding difference between Target Network and Double DQN From this blog: Target Network generates the target-Q values that will be used to compute the loss for every ...
5
votes
1answer
3k views

Simple Q-Table Learning: Understanding Example Code

I'm trying to follow a tutorial for Q-Table learning from this source, and am having difficulty understanding a small piece of the code. Here's the entire block: ...
5
votes
1answer
1k views

What is Compatible Function Approximation theorem in reinforcement learning?

I am following David Silver's RL course. In the policy gradient section, I found this slide that I would like have an explanation of. What are these two conditions? What is the logic behind the ...
5
votes
1answer
209 views

RL's policy gradient (REINFORCE) pipeline clarification

I try to build a policy gradient RL machine, and let's look at the REINFORCE's equation for updating the model parameters by taking a gradient to make the ascent (I apologize if notation is slightly ...
5
votes
1answer
7k views

What are the advantages / disadvantages of off-policy RL vs on-policy RL?

There are various algorithms for reinforcment learning (RL). One way to group them is by "off-policy" and "on-policy". I've heard that SARSA is on-policy, while Q-Learning is off-policy. I think they ...
5
votes
1answer
193 views

Is feature scaling necessary in reinforcement learning for the agent to learn successfully?

I often have trouble deciding how to tweak my input data for the agent. I am changing my data so that values in the hundreds get changed to a value between -1 and 1, and values that are originally ...
5
votes
1answer
237 views

how to choose between discounted reward and average reward

how to select between average reward and discounted reward? And when average reward is more effective in comparison with discounter reward and when vice versa is correct? -Is is possible to use both ...
4
votes
2answers
119 views

Is reseating passengers a reinforcement learning problem?

Requirement is to optimally move passengers from one seat map to another which has a different configuration. Move should be based on many rules like - 1) Families should be sitting together 2) Those ...
4
votes
1answer
113 views

selecting a number of neurons specifically for RL

If I would like to use DQN to train my Reinforcement-Learning agent, how do I select the number of neurons? In Supervised Learning, selecting too few or too many can result in either a too low ...
4
votes
2answers
1k views

Information extraction with reinforcement learning, feasible?

I was wondering if one could use Reinforcement Learning (as it is going to be more and more trendy with the Google DeepMind & AlphaGo's stuff) to parse and extract information from text. For ...
4
votes
1answer
798 views

What is “Policy Collapse” and what are the causes?

I saw the term "policy collapse" on the comments of a tutorial for reinforcement learning. I'm guessing that it's referred to as a policy collapse when the policy worsens over training due to a bad ...
4
votes
1answer
277 views

Large action space for deep reinforcement learning

I know that in normal Deep Reinforcement Learning(DRL) scenario, we learn a deep neural network to map current states to Q values. The number of the Q values (# of outputs of the neural network) is ...
4
votes
2answers
614 views

Hindsight Experience Replay, how to define a partially-known End-Goal

One of the requirements of the Hindsight Experience Replay is supplying the DQN with a state and a goal (the desired end-state) that we hope to end up in: ...
4
votes
2answers
3k views

what is difference between the DDQN and DQN?

I think I did not understand what is the difference between DQN and DDQN in implementation. I understand that we change the traget network during the running of DDQN but I do not understand how it is ...
4
votes
1answer
843 views

How does Implicit Quantile-Regression Network (IQN) differ from QR-DQN?

For several months I browsed the internet hoping to find a user-friendly explanation of the Implicit Quantile Regression Network (IQN). But, it seems there is none at all. How does IQN differ from ...
4
votes
1answer
150 views

How can RL agents be monitored?

My question is about how to monitor RL agents in production. To make the question easier to discuss, here is a use case. Please don't focus on difficulties in implementing such an agent, but rather on ...
4
votes
1answer
495 views

Parallel Q-learning

I'm looking for academic papers or other credible sources focusing on the topic of parralelized reinforcement learning, specifically Q-learning. I'm mostly interested in methods of sharing Q-table ...
4
votes
1answer
320 views

How differential semi-gradient Sarsa updates estimated average reward?

I cannot understand the way how algorithm Differential Semi-gradient Sarsa updates its estimated average reward $\bar{R}$. The algorithm I am looking at is from Sutton's text book Reinforcement ...