Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Filter by
Sorted by
Tagged with
3
votes
1answer
435 views

Rainbow vs A3C …too unfair?

In Deep Mind's Rainbow paper, how come A3C algorithm be so slow? twice slower than DDQN... Was it trained on a single actor? :D It's on page 1 of the paper Wasn't A3C supposed to be something a lot ...
4
votes
1answer
230 views

selecting a number of neurons specifically for RL

If I would like to use DQN to train my Reinforcement-Learning agent, how do I select the number of neurons? In Supervised Learning, selecting too few or too many can result in either a too low ...
11
votes
2answers
3k views

Prioritized Replay, what does Importance Sampling really do?

I can't understand the purpose of importance-sampling weights (IS) in Prioritized Replay (page 5). A transition is more likely to be sampled from experience replay the larger its "cost" is. ...
3
votes
0answers
105 views

Deep advantage learning: how to predict the value

I'm currently working on a collection of reinforcement algorithms: https://github.com/lhk/rl_gym For deep q-learning, you need to calculate the q-values that should be predicted by your network. ...
1
vote
1answer
382 views

Reframing action recognition as a reinforcement learning problem

Given the significant advancements in reinforcement learning, I wanted to know whether it is possible to recast problems such as action recogniton, object tracking, or image classification into ...
1
vote
1answer
247 views

Dueling DQN - Advantage Stream, why use average and not the tanh?

For Dueling DQN (page 5), why do authors use an average for Advantage stream, and don't simply "activate" the Advantage stream (with a $tanh$ for example)? Would "activating" work in theory, and is ...
2
votes
1answer
214 views

Dueling DQN what does a' mean?

what does $a'$ mean in the "combining" equation in Dueling DQN? (top of the page 5) $$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(...
8
votes
2answers
3k views

How does generalised advantage estimation work?

I've been trying to add GAE to my A2C implementation for a while now, but I' can't quite seem to grok how it works. My understanding of it, is that it reduces the variance of the advantage estimation ...
4
votes
2answers
717 views

Prioritized Experience Replay - why to approximate the Density Function?

I am reading about Prioritized Experience Replay, and can't understand the following: On page 4, every transition can be selected from the table with its own probability. Here is the cumulative ...
1
vote
0answers
385 views

Implementing experience replay in reinforcement learning

I've been reading Google's DeepMind Atari paper and I'm trying to understand how to implement experience replay. My question is that whether we update the ...
6
votes
1answer
5k views

Q-Learning: Target Network vs Double DQN

I am having a hard time understanding difference between Target Network and Double DQN From this blog: Target Network generates the target-Q values that will be used to compute the loss for every ...
2
votes
1answer
452 views

Reinforcement Learning in 2018, best tips and tricks?

Putting aside things applicable to neural networks such as dropout, l2 regularization, new opitmizers - what are the cool things I should be adding to my Reinforcement Learning algorithm (Q-Learning ...
2
votes
0answers
557 views

Defining State Representation in Deep Q-Learning

So I am having difficulty difficulty figuring out exactly how I want to represent my environment state in my Deep Q-learning problem. Premise: There is a 2D grid space of which an agent needs to ...
3
votes
2answers
55 views

Supervised Learning could be biased if we use obsolete data [closed]

What if the data that we could use for the training is obsolete. For instance, if I train my model with the computer sales report from the 20th century and try to predict the actual trends, a disaster,...
1
vote
1answer
42 views

Adding a bias makes Q-learning algorithm ineffective

I've been working through the Q-Network learning example in this Arthur Juliani's blog. It's based on the pretty trivia Open Gym Frozen Lake example. It's base implementation get's about 47% success ...
3
votes
1answer
128 views

Choosing a right algorithm for template-based text generation

I am doing a text generation project -- the task is to basically represent the statistical data in a readable way. The way I decided to go about this is template-based: each data type has a template ...
0
votes
1answer
590 views

Why are policy gradients on-policy?

I'm not entirely sure why policy gradients have to be on-policy and have to update using trajectories sampled from the current behaviour. In REINFORCE, the loss function is determined by the log ...
1
vote
1answer
65 views

Why infinite sampling is not realisitc assumpition in most real applications

I came across the below paragraphs, which I believe are the answers to the question Why infinite sampling is not realistic assumption in most real applications. Still i dont get the below explanation ?...
3
votes
2answers
1k views

What is the difference between bootstrapping and sampling in reinforcement learning?

I have come across a David Silver's slide which contains both the terms "bootstrapping" and "sampling". Is there any realistic example which helps me to understand the concepts better.
0
votes
1answer
29 views

Is there any reason why providing symbolic features into an MLP wouldn't outperform feeding raw pixels to a CNN in a RL task?

I am tackling a RL problem (relaxed version of Space Fortress) with DQN. The usual approach would be to feed pixels into a CNN but that is usually very slow. I am considering feeding symbolic features ...
1
vote
1answer
692 views

Formulate a MDP for a problem given below

Can someone help me to formulate an MDP for the below problem ?. Problem Definition Bunny wakes up in a strange room with 2 doors; one on the left, and one on the right. In front of him is a map of ...
0
votes
1answer
199 views

Can you interpolate with QLearning or Reinforcement learning in general?

I am currently researching the usages of machine learning paradigms for pathfinding problems. I am currently looking into the reinforcement learning paradigm and I used QLearning for pathfinding. ...
3
votes
1answer
67 views

What is the optimal value of a Markov Decision process with Single actions at each state?

I am trying to solve some questions about a MRP (i.e. a Markov Decision process with only one possible action at each state). The setup is as follows: There are two states ($a$ and $b$) stepping to $...
6
votes
2answers
3k views

RL Policy Gradient: How to deal with rewards that are strictly positive?

In short: In the policy gradient method, if the reward is always positive (never negative), the policy gradient will always be positive, hence it will keep making our parameters larger. This makes ...
1
vote
1answer
49 views

Policy gradient on data only, without emulators

It is too costly for my team to emulate the agent (executing the action and assessing the reward), meaning our only option is to learn the optimal policy on our dataset. The good thing is that we have ...
1
vote
1answer
195 views

Reinforcement Learning with static state

Can Q Learning work with a static state for each step? What I mean by that is that the actions do not influence the following state at all. The episodes just iterate over the same data over and over ...
1
vote
1answer
838 views

What are the practical applications of inverse reinforcement learning? [closed]

Inverse reinforcement learning is about using expert trajectories to learn a reward function. Now the most successful method is Maximum Entropy Inverse Reinforcement Learning. But in that, you need a ...
1
vote
1answer
55 views

Need help in deriving Policy Evaluation (Prediction)

Policy Evaluation is computing the state-value function for an arbitary policy $\pi$.(suton & barto book). Now \begin{equation} v_{\pi} = E_{\pi}[\;G_{t}\,|\;S_{t}=s] \qquad\qquad\qquad\qquad(4....
8
votes
2answers
1k views

Is this a Q-learning algorithm or just brute force?

I have been playing with an algorithm that learns how to play tictactoe. The basic pseudocode is: ...
1
vote
1answer
435 views

Card game for Gym: Reward shaping

I am working on a card game for openai gym and currently I ask myself how to shape the reward function for it. One round of the game consists of each player picking a card from its hand, whereas not ...
0
votes
1answer
152 views

Reinforcement Learning (Q Learning)

I was reading a paper on traffic flow optimization using Multi-Agent Q learning. the paper proposes the following method: Deploy a Reinforcement learning controller at each intersection with traffic ...
0
votes
1answer
784 views

Reinforcement Deep Learning for object detection [closed]

After reading the state of the Art of object detection using the CNN's(R-CNN,Faster R-CNN,YOLO,YOLOv2,SSD) I was wondering if there is an efficient method that use deep learning with reinforcement ...
3
votes
1answer
625 views

Why Deep Reinforcement Learning fails to learn how to play Asteroids?

Deep Q-learning, A3C, policies evolved with genetic algorithms, they all fail to learn Asteroids, or at least perform way worse than human. From the hardest Atari games according to RL most of the ...
1
vote
0answers
29 views

Does employment of engineered immediate rewards in RL introduce a non-linear problem to an agent?

Suppose we operate with state-action pairs called 'S', and a reward function R() as follows: ...
0
votes
1answer
170 views

Multivariable real time system for fraud detection

This is the scenario: Client -> Server The client sends multiple voice calls to Server. Call info: calling number called number call duration source IP ...
1
vote
1answer
175 views

Clarifying my understanding of on-policy RL (online SARSA)

I want to clarify I have understood how SARSA works in nuances. Consider an original definition taken from ON-LINE Q-LEARNING USING CONNECTIONIST SYSTEMS. G. A. Rummery & M. Niranjan. CUED/F-...
3
votes
1answer
2k views

Reinforcement Learning on data only (NO emulators)

My team and I started digging into RL for the purpose of a specific application. We have plenty of data of an agent carrying out suboptimal policies (states and rewards...). It is too costly for us ...
2
votes
2answers
586 views

A3C - Turning action probabilities into intensities

I'm experimenting with using an A3C network to learn to play old Atari video games. My network outputs a set of probabilities for each possible action (e.g. left, right, shoot), and I use this ...
5
votes
2answers
3k views

What is a policy in machine learning?

While I was reading the paper "Grounded Action Transformation for Robot Learning in Simulation", I came across the term "policy". Could someone explain to me what that actually is (in general and in ...
1
vote
0answers
188 views

Alternatives to Markov decision processes in reinforcement learning

In reinforcement learning, if the Markov property holds, then an environment defines or can be modeled as a Markov decision process (MDP). A finite MDP is an MDP with finite state and action sets. ...
26
votes
2answers
9k views

What exactly is bootstrapping in reinforcement learning?

Apparently, in reinforcement learning, temporal-difference (TD) method is a bootstrapping method. On the other hand, Monte Carlo methods are not bootstrapping methods. What exactly is bootstrapping ...
1
vote
1answer
162 views

Understanding why in deep reinforcement learning correlations in the data reduce the effectiveness

From the paper Human-level control through deep reinforcement learning, Mnih et al. Nature 2015 It says ...
3
votes
0answers
136 views

Graphical results of Q-Learning: is improvement possible by parameter tweaking?

From left to right: Maximum Q value for action selection (averaged) Train error (averaged) Reward from environment (averaged) I run double Q-learning. A behavioral policy is ε-greedy, ε constant ...
1
vote
1answer
593 views

Q Learning Neural network for tic tac toe Input implementation problem

I've recently become interested in machine learning, specifically neural networks, and after creating ones to solve basic problems such as XOR and Sin and Cos graphs, however i am now looking into ...
1
vote
0answers
43 views

An unbiased simulator for policy simulation in reinforcement learning

I'm reading the following paper https://arxiv.org/pdf/1003.0146.pdf on building a fair online simulator for contextual bandits. In particular i'm interested in the proof on page 665 of Theorem 1 I ...
1
vote
0answers
94 views

What are the major differences between Facebook's ELF RF framework and TensorFlows “Agents”?

I recently read a few papers, especially ELF: An Extensive, Lightweight, and Flexible Research Platform for Real-time Strategy Games TensorFlow Agents: Efficient Batched Reinforcement Learning in ...
3
votes
1answer
4k views

Is my understanding of On-Policy and Off-Policy TD algorithms correct?

After reading several questions here and browsing some pages on the topic, here is my understanding of the key difference between Q-learning (as an example of off-policy) and SARSA (as an example of ...
0
votes
1answer
403 views

contextual bandits for online learning

Which of the algorithms in the current literature for contextual bandits can be implemented for online learning and which ones can't? I'd really appreciate it if someone could provide a link to papers ...
2
votes
1answer
121 views

Reasoning for temporal difference update rule

In TD(0) learning where the value function is given by $V(s) = w^T\phi(s)$ where $w$ is a weight vector and $\phi$ is a feature map, the weight update is given by $w_{t+1} = w_{t} + \eta\delta_{t+1}\...
2
votes
0answers
169 views

Epoch greedy algorithm for contextual bandits

I'm reading the following paper on the epoch greedy algorithm for the contextual bandits problem. I have two questions http://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf I'm unsure ...

1
6 7
8
9 10