Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Filter by
Sorted by
Tagged with
5
votes
1answer
3k views

Simple Q-Table Learning: Understanding Example Code

I'm trying to follow a tutorial for Q-Table learning from this source, and am having difficulty understanding a small piece of the code. Here's the entire block: ...
5
votes
1answer
193 views

Is feature scaling necessary in reinforcement learning for the agent to learn successfully?

I often have trouble deciding how to tweak my input data for the agent. I am changing my data so that values in the hundreds get changed to a value between -1 and 1, and values that are originally ...
1
vote
1answer
721 views

Reinforcement Learning algorithm for Optimized Trade Execution

My question deals with the algorithm described in the paper: Reinforcement Learning for Optimized Trade Execution This paper uses reinforcement learning technique to deal with the problem of ...
3
votes
1answer
202 views

Catastrophic forgetting in linear semi-gradient RL agent?

I've been working through the Sutton + Barto RL text, implementing a number of the algos + running them in the OpenAI gym. One phenomenon that I seem to come across quite regularly is that agents who, ...
3
votes
1answer
2k views

Has the Random Forest algorithm ever been used in Reinforcement Learning applications?

I've seen a research paper describing a "Reinforcement Learning Tree", which the authors say that it has a better convergence than random forests. However, I couldn't find anything related to ...
4
votes
1answer
793 views

What is “Policy Collapse” and what are the causes?

I saw the term "policy collapse" on the comments of a tutorial for reinforcement learning. I'm guessing that it's referred to as a policy collapse when the policy worsens over training due to a bad ...
3
votes
1answer
572 views

Neural Network Learning Rate vs Q-Learning Learning Rate

I'm just getting into machine learning--mostly Reinforcement Learning--using a neural network trained on Q-values. However, in looking at the hyper-parameters, there are two that seem redundant: the ...
7
votes
2answers
2k views

Why does Q Learning diverge?

My Q-Learning algorithm's state values keep on diverging to infinity, which means my weights are diverging too. I use a neural network for my value-mapping. I've tried: Clipping the "reward + ...
-1
votes
1answer
43 views

How should values that “don't exist” sometimes be handled as input data?

I'm currently training an agent to learn how to fight in a shooting game. I'm using the bullet positions of the agent's opponent as one of the features. The features "don't exist" when the opponent ...
1
vote
0answers
497 views

Keras not converging to optimum while TensorFlow does

I'm working on a Reinforcement learning project where the agent needs to navigate itself around the maze and get to the goal. (I used Q Learning as my algorithm) The agent found the optimal path in 50 ...
0
votes
1answer
365 views

Q-learning with a state-action-state reward structure and a Q-matrix with states as rows and actions as columns

I have set up a Q-learning problem in R, and would like some help with the theoretical correctness of my approach in framing the problem. Problem structure For this problem, the environment consists ...
18
votes
1answer
28k views

What is “experience replay” and what are its benefits?

I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
3
votes
1answer
2k views

Reinforcement learning for continuous (rather than discrete) actions

I'm familiar with traditional reinforcement learning where the algorithm must choose a categorical action (e.g., best move in a game or the highest click-through-rate ad among a set of ads). Does ...
8
votes
2answers
6k views

Why do we normalize the discounted rewards when doing policy gradient reinforcement learning?

I'm trying to understand the policy gradient approach for solving the cartpole problem. In this approach, we're expressing the gradient of the loss w.r.t each parameter of our policy as an expectation ...
3
votes
1answer
541 views

Understanding the training phase of the tutorial “Using Keras and Deep Deterministic Policy Gradient to play TORCS” tutorial

I am trying to understand the training phase of the tutorial Using Keras and Deep Deterministic Policy Gradient to play TORCS (mirror, code) by Ben Lau published on October 11, 2016. The tutorial ...
1
vote
1answer
1k views

Semi-gradient TD(0) Choosing an Action

I am trying to write an optimal control agent for a simple game that looks like this: The agent can only move along the x-axis, and has three actions available to it: left, right, and do nothing. A ...
6
votes
1answer
621 views

Reward dependent on (state, action) versus (state, action, successor state)

I am studying reinforcement learning and I am working methodically through Sutton and Barto's book plus David Silver's lectures. I have noticed a minor difference in how the Markov Decision Processes ...
1
vote
1answer
127 views

Tensorflow / Deepmind: how do I take actions from observations for math algorithms related to proofs?

Crossposted from: https://stackoverflow.com/questions/42809054/tensorflow-deepmind-how-do-i-take-actions-from-observations-for-math-algorith This question is to ask for directions/suggestions/help on ...
1
vote
0answers
286 views

What is significance of Colour-digit MNIST game in paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning?

My question is regarding the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning (https://arxiv.org/abs/1605.06676). Can anyone explain what is the significance of Colour-digit ...
1
vote
2answers
541 views

Reinforcement learning, pendulum python

I'm having trouble finding a good reward function for the pendulum problem, the function I'm using: $-x^2 - 0.25*(\text{xdot}^2)$ which is the quadratic error from the top. with $x$ representing the ...
8
votes
3answers
657 views

What knowledge do I need in order to write a simple AI program to play a game?

I'm a B.Sc graduate. One of my courses was 'Introduction to Machine Learning', and I always wanted to do a personal project in this subject. I recently heard about different AI training to play games ...
1
vote
1answer
942 views

Advantage Function - Variance Reduction

When explaining advantage function, it is usually claimed that using a baseline reduces the variance. I have not found any specific reference to justify this. Is this an application of control ...
3
votes
1answer
395 views

Reinforcement learning: understanding this derivation of n-step Tree Backup algorithm

I think I get the main idea, and I almost understand the derivation except for this one line, see picture below: I understand what we're doing by using the policy probability to weight the rewards ...
5
votes
1answer
6k views

Understanding advantage functions

The paper explaining 'Advantage Updating' as a method to improve Q-learning uses the following as its motivation. Q-learning requires relatively little computation per update, but it is useful to ...
2
votes
1answer
661 views

What is the reward function in the 10 armed test bed?

The Sutton & Barto book on reinforcement learning mentions the 10 armed test bed in chapter 2, Bandit Problems: To roughly assess the relative effectiveness of the greedy and ε-greedy methods,...
2
votes
0answers
780 views

Initial Q-values in Q-Learning

I am running a Q-learning algorithm with a finite time horizon. Are 'optimistic initial conditions' still preferred if there is a possibility that some states will not be visited multiple times? ...
6
votes
5answers
476 views

What is Reinforcement Learning?

I am familiar with the concepts of Supervised and Unsupervised Learning but recently Reinforcement (Reinforced?) Learning also popped up before me a couple of times. Could anyone give a hint what is ...
1
vote
0answers
139 views

Policy network AlphaGo and transferring to other domains

This is not covered very well in their AlphaGo paper but I assume that their policy network has a softmax output layer with a node for all the positions on the board, including illegal ones (the one ...
1
vote
0answers
111 views

What did DeepMind do with AlphaGo between the Fan Hui and Lee Sedol games?

In January, DeepMind published the article (see video) about its win against Fan Hui, which happend in October 2015. The article and other interviews say, it used 100.000 human games, and then 13.000....
5
votes
1answer
7k views

What are the advantages / disadvantages of off-policy RL vs on-policy RL?

There are various algorithms for reinforcment learning (RL). One way to group them is by "off-policy" and "on-policy". I've heard that SARSA is on-policy, while Q-Learning is off-policy. I think they ...
3
votes
2answers
369 views

Is there some model-based variation of the Q-Learning algorithm which learns on a 3D SxAxS' table instead of a 2D SxA table?

Q-learning works with a 2D SxA table of Q values, where S is the current state and A is the action taken. Is there some model-based variant of Q-learning (or SARSA) that uses a 3D SxAxS' table to ...
12
votes
1answer
3k views

Supervised learning vs reinforcement learning for a simple self driving rc car

I'm building a remote-controlled self driving car for fun. I'm using a Raspberry Pi as the onboard computer; and I'm using various plug-ins, such as a Raspberry Pi camera and distance sensors, for ...
14
votes
4answers
452 views

AlphaGo (and other game programs using reinforcement-learning) without human database

I am not a specialist of the subject, and my question is probably very naive. It stems from an essay to understand the powers and limitation of reinforcement learning as used in the AlphaGo program. ...
25
votes
5answers
13k views

Difference between AlphaGo's policy network and value network

I was reading a high level summary about Google's AlphaGo (http://googleresearch.blogspot.co.uk/2016/01/alphago-mastering-ancient-game-of-go.html), and I came across the terms "policy network" and "...
1
vote
1answer
863 views

Combining Neural Network with Reinforcement Learning in a Continuous Space

I'm trying to learn how to do reinforcement learning on my own and I am not sure how to implement a neural network for a specific problem. The game goes on for roughly 1 million steps. At each step, ...
4
votes
2answers
1k views

Information extraction with reinforcement learning, feasible?

I was wondering if one could use Reinforcement Learning (as it is going to be more and more trendy with the Google DeepMind & AlphaGo's stuff) to parse and extract information from text. For ...
7
votes
2answers
786 views

What is the novelty in AlphaGo, Google Deepmind's Go playing system?

Recently researchers at Google DeepMind published a paper, where they described a Go playing system that beat the best current computer programs and the human European champion. I had a quick look at ...
30
votes
5answers
23k views

What is the Q function and what is the V function in reinforcement learning?

It seems to me that the $V$ function can be easily expressed by the $Q$ function and thus the $V$ function seems to be superfluous to me. However, I'm new to reinforcement learning so I guess I got ...
4
votes
1answer
492 views

Parallel Q-learning

I'm looking for academic papers or other credible sources focusing on the topic of parralelized reinforcement learning, specifically Q-learning. I'm mostly interested in methods of sharing Q-table ...
7
votes
2answers
3k views

How to teach neural network a policy for a board game using reinforcement learning?

I need to use reinforcement learning to teach a neural net a policy for a board game. I chose Q-learining as the specific alghoritm. I'd like a neural net to have the following structure: layer - <...
6
votes
4answers
1k views

Does reinforcement learning require the help of other learning algorithms?

Can't reinforcement learning be used without the help of other learning algorithms like SVM and MLP back propagation? I consulted two papers: Paper 1 Paper 2 both have used other machine learning ...
2
votes
1answer
99 views

Value Updation Dynamic Programming Reinforcement learning

Regarding Value Iteration of Dynamic Programming(reinforcement learning) in grid world, the value updation of each state is given by: Now Suppose i am in say box (3,2). I can go to (4,2)(up) (3,3)(...
0
votes
1answer
482 views

Confusion in Policy Iteration and Value iteration in Reinforcement learning in Dynamic Programming

What I understood for value iteration while coding is that we need to have a policy fixed. According to that policy the value function of each state will be calculated. Right? But in policy iteration ...
3
votes
1answer
131 views

When to stop calculating values of each cell in the grid in Reinforcement Learning(dynamic programming) applied on gridworld

Considering application of Reinforcement learning(dynamic programming method performing value iteration) on grid world, in each of the iteration, I go through each of the cell of the grid and update ...
9
votes
4answers
2k views

Books on Reinforcement Learning

I have been trying to understand reinforcement learning for quite sometime, but somehow I am not able to visualize how to write a program for reinforcement learning to solve a grid world problem. Can ...
10
votes
2answers
292 views

Cooperative Reinforcement Learning

I already have a functioning $Q(\lambda)$ implementation for a single agent working on a dynamic pricing problem with the goal of maximizing revenue. The problem that I'm working with, however, ...
3
votes
1answer
701 views

learning rate in reinforcement learning

Does anyone know how to get the learning rate from participant data? I'm computing all the expected values for all trials (=200) $$V_t(S) = V_{t-1}(S)+ \alpha \cdot \text{error }_t $$ $$(\text{...
0
votes
1answer
2k views

How to generate ratings without training data?

I am working on generating restaurant ratings automatically and I have various feature values like delivery time, cost estimate, etc. I want to generate a rating for each restaurant between 0 to 5. ...
1
vote
2answers
456 views

Does reinforcement learning only work on grid world?

Does reinforcement learning always need a grid world problem to be applied to? Can anyone give me any other example of how reinforcement learning can be applied to something which does not have a ...
10
votes
2answers
455 views

implementing temporal difference in chess

I have been developing a chess program which makes use of alpha-beta pruning algorithm and an evaluation function that evaluates positions using the following features namely material, kingsafety, ...