Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Filter by
Sorted by
Tagged with
0
votes
1answer
16 views

In first visit monte carlo are we assuming the environment is the same over episodes?

Watching this video (11:30) that presents the simplest algorithm for reinforcement learning: Monte Carlo Policy Evaluation, which says in general: The first time a sate is visited: increment N(s): N(...
1
vote
1answer
97 views

Reinforcement Learning - How are these state values in MRP calculated?

This is a question from the book an Introduction to RL, page 125, example 6.2. The example compares the prediction abilities of TD(0) and constant $ \alpha $ MC when applied to the below Markov ...
4
votes
1answer
838 views

How does Implicit Quantile-Regression Network (IQN) differ from QR-DQN?

For several months I browsed the internet hoping to find a user-friendly explanation of the Implicit Quantile Regression Network (IQN). But, it seems there is none at all. How does IQN differ from ...
1
vote
1answer
45 views

TD Learning formula

This is something I cannot get my head around and initially I thought is a typo but it is not. Essentially in TD learning, we are trying to learn the Value Function. A value function tells me how ...
1
vote
1answer
1k views

IndexError: index 804 is out of bounds for axis 0 with size 800

i installed a self driving car project from superdatascience site , when i open the map using terminal after a while the map window close up or it closes directly after i maximize the map window and ...
0
votes
1answer
34 views

Do we need to use off-policy methods for policy shaping?

Let's say that there is a reinforcement learning task and an agent in a environment. I want a human teacher to manually modify the policy of the agent (policy shaping) to speed up the learning of the ...
1
vote
1answer
50 views

Will reinforcement learning work if states wont get repeated again?

I am working on a information retrieval model where the user enters a query and the model has to retrieve 3 most relevant FAQ pairs.I am collecting implicit feedback in terms of page clicks etc.What I ...
1
vote
0answers
91 views

Dueling DQN - Calculation of Q-value

I'm trying to implement a Double Dueling DQN on LunarLander and I'm facing an issue as my model is not learning so I'm trying to debug the graph and this leads me to a question regarding the ...
1
vote
1answer
36 views

objective in policy gradient equation?

I don't understand how this was deduced from first equation to second expectation. Is it from conditional probability theory? I checked but still can't understand. From wikipedia, the expectation of a ...
1
vote
1answer
59 views

Stability of value function approximation in policy gradients

In DQNs, function approximation of the Q-values is unstable for correlated updates. In policy gradients with a baseline, will the value function of the policy not be plagued by the same correlated ...
0
votes
1answer
150 views

What is “GOAL” in terms of Reinforcement Learning specified in these papers?

I have a question regarding Reinforcement Learning. I've been reading the Horde and the UVFA paper extensively. Take the Horde paper, there is this GVF, General Value Function Approximators which ...
2
votes
1answer
61 views

Is RL applicable to environments that are totally RANDOM?

I have a fundamental question on the applicability of reinforcement learning (RL) on a problem we are trying to solve. We are trying to use RL for inventory management - where the demand is entirely ...
1
vote
0answers
28 views

Deep RL: Visualizing/Analyzing the gradient

I am testing different RL methods, and I know e.g that policy gradient method is supposed to have a high variance gradient which can cause trouble. I want to run a few different Deep RL algorithms, ...
1
vote
1answer
228 views

What is the immediate reward in value iteration?

Suppose you're given an MDP where rewards are attributed for reaching a state, independently of the action. Then when doing value iteration: $$ V_{i+1} = \max_a \sum_{s'} P_a(s,s') (R_a(s,s') + \...
2
votes
1answer
92 views

Reinforcement learning: easily learnable state representation

I have created a simple OpenAI Gym environment, which consists of: A continuous 2D world with x and y in range [0.0, 1.0] A rabbit which slowly moves randomly in the world with a constant speed A '...
2
votes
1answer
177 views

What are the differences between Reinforcement Learning (RL) and Supervised Learning?

What is the difference between Reinforcement Learning (RL) and Supervised Learning? Does RL hava more difficulty in finding a stable solution? Does Q-learning have more difficulty in finding a ...
1
vote
0answers
476 views

DQN cannot learn or converge

I have implemented a DQN using keras. The task is to collect the circles and avoid the red circle and crosses. The associated rewards are +5, -5 and 0 otherwise. if the agent go out of the board, the ...
1
vote
1answer
3k views

What is the relationship between MDP and RL?

What is the relationship between Markov Decision Processes and Reinforcement Learning? Could we say RL and DP are two types of MDP?
1
vote
0answers
43 views

Reinforcement learning - How to deal with varying number of actions which do number approximation

I am a new to Reinforcement learning, but I am trying to use RL in this task: Given a function definition in written e.g. in C with 1 to 10s of input arguments (only numerical ones - integer, float, ...
2
votes
1answer
464 views

What are features in the context of reinforcement learning?

In machine learning, "feature" is a synonym for explanatory variables. I know what a feature is. However, in the specific case of RL, it's not clear to me what features are. What are "features" in the ...
4
votes
2answers
3k views

what is difference between the DDQN and DQN?

I think I did not understand what is the difference between DQN and DDQN in implementation. I understand that we change the traget network during the running of DDQN but I do not understand how it is ...
1
vote
1answer
72 views

Boundaries of Reinforcement Learning

I finally developed a Game Bot that learns how to play the videogame Snake with Deep Q-Learning. I tried with different neural networks and hyper-parameters, and I found a working set-up, for a ...
5
votes
1answer
208 views

RL's policy gradient (REINFORCE) pipeline clarification

I try to build a policy gradient RL machine, and let's look at the REINFORCE's equation for updating the model parameters by taking a gradient to make the ascent (I apologize if notation is slightly ...
1
vote
1answer
177 views

Should reinforcement learning always assume (PO)MDP?

I recently just started learning reinforcement learning and learned that reinforcement learning algorithms work under the assumption of MDP or POMDP. However as I read A3C and recent vision based deep ...
1
vote
0answers
253 views

Policy gradient: why does this converge with Adam and not SGD?

I am looking into policy gradient methods. I stumbled into this implementation: https://gist.github.com/calclavia/cfcd41ad4e47d7b9b6ab8af15410747a It uses a Nesterov Adam optimizer. If I run it, it ...
3
votes
2answers
231 views

Reinforcement learning: Discounting rewards in the REINFORCE algorithm

I am looking into the REINFORCE algorithm for reinforcement learning. I am having trouble understanding how rewards should be computed. The algorithm from Sutton & Barto: What does G, 'return ...
0
votes
1answer
46 views

Can RL learn to mimic a hash function?

Theoretically, is this possible: When you have a given input string, there are a set of permutations and bit operations to do. The problem is, when you choose an approach like reinforcement learning, ...
1
vote
2answers
42 views

Equations in “Intoduction to RL”: What is the meaning and difference between E, and E with subscript?

This question is from An introduction to RL, page 78. In the formula below the page, both $\mathbb{E}$ and $\mathbb{E_\pi}$ are mentioned. Could you help me understand the difference between ...
1
vote
1answer
330 views

Dueling DQN - why should we decompose and then combine them back into?

could anyone who can help me if we decompose them and combine back them into a single Q, what the network can learn? from my perspective,the V means the total reward when the agent follow the current ...
0
votes
1answer
126 views

Policy Gradient Methods - ScoreFunction & Log(policy)

In Policy Gradient Methods, Lecture 7 (34:15), David describes a Score Function as being the Gradient of the Log of the policy Question: If we have a Neural ...
5
votes
1answer
2k views

Reinforcement learning: decreasing loss without increasing reward

I'm trying to solve OpenAI Gym's LunarLander-v2. I'm using the Deep Q-Learning algorithm. I have tried various hyperparameters, but I can't get a good score. Generally the loss decreases over many ...
2
votes
1answer
63 views

Optimization motion planning by using Bellman equation

From the Montana article "Kinematics of Contact and Grasp", if I have a ball roll on the plane without sliding, the motion equation is described below: \begin{equation*} \begin{bmatrix} \dot{u}_{2} \\ ...
2
votes
1answer
3k views

RL Advantage function why A = Q-V instead of A=V-Q?

In RL Course by David Silver - Lecture 7: Policy Gradient Methods, David explains what an Advantage function is, and how it's the difference between Q(s,a) and the V(s) Preliminary, from this post: ...
2
votes
1answer
63 views

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )? I want to replace the DNN section of Qlearning with a RF or SVR but the problem is that there is no clear ...
1
vote
1answer
106 views

Is the neural network in DQN used to learn like a supervised model?

Is the neural network in DQN used to learn like a supervised model?
3
votes
1answer
132 views

How to use a different model to deep neural network with reinforcement learning based on DQN?

Is it possible to implement a reinforcement learning algorithm without using a deep neural network (DNN) as used in deep reinforcement learning e.g. Deep Q-Network (DQN)? How can I replace the DNN in ...
1
vote
1answer
75 views

MDP - RL, Multiple rewards for the same state possible?

This question is from An introduction to RL Pages 48 and 49. This question may also be related to below question, although I am not sure: Cannot see what the "notation abuse" is, mentioned ...
9
votes
1answer
4k views

Can Reinforcement learning be applied for time series forecasting?

Can Reinforcement learning be applied for time series forecasting?
2
votes
1answer
247 views

RL agent, how to forbid actions?

In Q-learning, how to tell the agent that action $a_7$ is unavailable from within state $s_{t}$? Is supplying a very large negative reward good, or might throw it off-track? From what I read (link),...
3
votes
1answer
155 views

Time horizon T in policy gradients (actor-critic)

I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture. At the bottom of that slide, the gradient of the expected sum of rewards ...
1
vote
0answers
137 views

Potential-based reward shaping in DQN reinforcement learning

I work for quite some time on a RL task which poses a surprising difficulty to the reinforcement learning agent to learn. My technique is based on the Double DQN, with replay buffer, using recurrent ...
2
votes
1answer
649 views

Policy-based RL method - how do continuous actions look like?

I've read several times that Policy-based RL methods can work with continious action space, rather than with discrete actions (move left 5 meters, move right 5.5 meters), like Value-based methods (Q-...
1
vote
0answers
533 views

Why is my loss function for DQN converging too quickly?

I'm still relatively new to deep learning and am experiencing an issue that I can't seem to find a solution/explanation for. I've developed a DQN model in tensorflow, as described by DeepMind, and am ...
2
votes
1answer
139 views

Importance Sampling in Off-policy n-step Sarsa

In Chapter 7.3 of Reinforcement Learning: An Introduction by Sutton and Barto, the off-policy pseudocode has the following update equation for $Q$: Compute importance sampling ratio: $$ \rho \...
2
votes
1answer
146 views

What is wrong with this reinforcement learning environment ?

I'm working on below reinforcement learning problem: I have bottle of fix capacity (say 5 liters). At the bottom of bottle there is cock to remove water. The distribution of removal of water is not ...
1
vote
1answer
290 views

How to give rewards to actions in RL?

I'm working on below reinforcement learning problem: I have bottle of fix capacity (say 5 liters). At the bottom of bottle there is cock to remove water. The distribution of removal of water is not ...
1
vote
0answers
184 views

Reinforcement algorithm for binary classification

I am new to machine learning, but I've read a lot about Reinforcement Learning in the past 2 days. I have an application that fetches a list of projects (e.g. from Upwork). There is a moderator that ...
1
vote
0answers
207 views

Deep Q-Learning with large number of actions

I'm using DQN with large number of actions in [0, 10000, step = 1000]. This means I have an action space of size 11 (including 0 and 10000). Action space is still discrete. My problem is that, instead ...
3
votes
1answer
312 views

Difference between advantages of Experience Replay in DQN2013 paper

I've been re-reading the Playing Atari with Deep Reinforcement Learning (2013) paper. It lists three advantages of experience replay: This approach has several advantages over standard online Q-...
1
vote
1answer
727 views

Hindsight Experience Replay: what the reward w.r.t. to sample goal means

Referring to the paper on Hindsight Experience Replay Is it right that sampled goals which are visited states should be followed by a positive (or non-negative) rewards in order to allow an agent ...