Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

0
votes
1answer
11 views

In Reinforcement Learning can I randomly assign next_states from the state space to my agent while creating transition set?

In Reinforcement Learning, while creating transition samples (state, action, next_state, reward), where: Agent: The learning agent Environment: The trainer The environment gives two feedback to the ...
1
vote
1answer
26 views

Reinforcement learning: negative reward (punish) illegal actions?

If you train an agent using reinforcement learning (with Q-function in this case), should you give a negative reward (punish) if the agent proposes illegal actions for the presented state? I guess ...
0
votes
0answers
31 views

Reinforcement learning using Restricted Boltzmann Machine(RBM)

I am very new to the field of machine learning, AI and related things. I am right now, in particular looking for some references where RBMs can be used to achieve Reinforcement learning, something ...
0
votes
0answers
12 views

Policy gradient - and auto-differentiation (Pytorch/Tensorflow)

In policy gradient, we have something like this: Is my understanding correct that if I apply log cross-entropy on the last layer, the gradient will be automatically calculated as per formula above?
1
vote
1answer
20 views

Does policy optimization learn policies to make better actions with higher probability? [closed]

When I talk about policy optimization, it is referred to the following picture, and it is linked to DFO/Evolution plus Policy Gradients. I would like to know is it correct to say: Policy ...
0
votes
0answers
7 views

In first visit monte carlo are we assuming the environment is the same over episodes?

Watching this video (11:30) that presents the simplest algorithm for reinforcement learning: Monte Carlo Policy Evaluation, which says in general: The first time a sate is visited: increment N(s): N(...
0
votes
0answers
10 views

Natural actor-critic with Q function approximation

Referring to some useful readings, for example, this one: lectures on reinforcement learning, I try to program a natural gradient actor-critic, following this nice property: $$\nabla_\theta J(\theta) ...
0
votes
0answers
24 views

Q-Learning: Dynamic alpha vs fixed alpha

I guess this is a basic question, but I have seen that instead of fixing the value of $\alpha\in[0, 1]$ when updating the Q-value, some set it to $\alpha=1/k$, where $k$ is the number of times a state-...
0
votes
0answers
12 views

Minimize policy change

I am doing some Reinforcement Learning and trying to minimize/limit policy change. That is, implement the following loss function: $L=\min_\theta (\pi-\pi_{prev})$ where $\pi$ and $\pi_{prev}$ are ...
0
votes
0answers
70 views

Reinforcement Learning for Cart Pole problem in OpenAI Gym

I have a very specific question about mapping the Observations to states in a POMDP problem. Once I get there, I can construct a Q-table with states as rows and actions as columns and use Q-Learning. ...
1
vote
1answer
30 views

Reinforcement Learning - How are these state values in MRP calculated?

This is a question from the book an Introduction to RL, page 125, example 6.2. The example compares the prediction abilities of TD(0) and constant $ \alpha $ MC when applied to the below Markov ...
2
votes
1answer
65 views

How does Implicit Quantile-Regression Network (IQN) differ from QR-DQN?

As a newbie, for several months I browsed the internet hoping to find a user-friendly explanation of the Implicit Quantile Regression Network (IQN). But, it seems there is none at all. How does IQN ...
1
vote
1answer
29 views

TD Learning formula

This is something I cannot get my head around and initially I thought is a typo but it is not. Essentially in TD learning, we are trying to learn the Value Function. A value function tells me how ...
0
votes
1answer
33 views

IndexError: index 804 is out of bounds for axis 0 with size 800

i installed a self driving car project from superdatascience site , when i open the map using terminal after a while the map window close up or it closes directly after i maximize the map window and ...
0
votes
1answer
27 views

Do we need to use off-policy methods for policy shaping?

Let's say that there is a reinforcement learning task and an agent in a environment. I want a human teacher to manually modify the policy of the agent (policy shaping) to speed up the learning of the ...
1
vote
1answer
23 views

Will reinforcement learning work if states wont get repeated again?

I am working on a information retrieval model where the user enters a query and the model has to retrieve 3 most relevant FAQ pairs.I am collecting implicit feedback in terms of page clicks etc.What I ...
1
vote
0answers
25 views

Dueling DQN - Calculation of Q-value

I'm trying to implement a Double Dueling DQN on LunarLander and I'm facing an issue as my model is not learning so I'm trying to debug the graph and this leads me to a question regarding the ...
1
vote
1answer
24 views

objective in policy gradient equation?

I don't understand how this was deduced from first equation to second expectation. Is it from conditional probability theory? I checked but still can't understand. From wikipedia, the expectation of a ...
1
vote
1answer
22 views

Stability of value function approximation in policy gradients

In DQNs, function approximation of the Q-values is unstable for correlated updates. In policy gradients with a baseline, will the value function of the policy not be plagued by the same correlated ...
0
votes
0answers
17 views

Penalize Neural Network Common Output

To practice reinforcement learning, I have made a neural network class and have been trying to teach it to play a toy system of Pokemon. There are three types of pokemon, and each pokemon has access ...
0
votes
0answers
20 views

Resampling to get equal predictive power per observation

This is probably a thing I am just not searching for correctly, but essentially my idea is this: given some machine learning classification $C$ based on an input dataset $D$, certain observations in $...
0
votes
1answer
40 views

What is “GOAL” in terms of Reinforcement Learning specified in these papers?

I have a question regarding Reinforcement Learning. I've been reading the Horde and the UVFA paper extensively. Take the Horde paper, there is this GVF, General Value Function Approximators which ...
2
votes
1answer
47 views

Is RL applicable to environments that are totally RANDOM?

I have a fundamental question on the applicability of reinforcement learning (RL) on a problem we are trying to solve. We are trying to use RL for inventory management - where the demand is entirely ...
1
vote
0answers
18 views

Deep RL: Visualizing/Analyzing the gradient

I am testing different RL methods, and I know e.g that policy gradient method is supposed to have a high variance gradient which can cause trouble. I want to run a few different Deep RL algorithms, ...
1
vote
1answer
31 views

What is the immediate reward in value iteration?

Suppose you're given an MDP where rewards are attributed for reaching a state, independently of the action. Then when doing value iteration: $$ V_{i+1} = \max_a \sum_{s'} P_a(s,s') (R_a(s,s') + \...
2
votes
1answer
56 views

Reinforcement learning: easily learnable state representation

I have created a simple OpenAI Gym environment, which consists of: A continuous 2D world with x and y in range [0.0, 1.0] A rabbit which slowly moves randomly in the world with a constant speed A '...
1
vote
0answers
21 views

have deep reinforcement learning algorithms the unified behavior?

Could we say the behavior of different deep reinforcement learning algorithms is very similar for MDP and POMDP? Could we say DRL algorithms present a unified approach for finite-horizon, infinite ...
2
votes
1answer
48 views

What are the differences between Reinforcement Learning (RL) and Supervised Learning?

What is the difference between Reinforcement Learning (RL) and Supervised Learning? Does RL hava more difficulty in finding a stable solution? Does Q-learning have more difficulty in finding a ...
1
vote
0answers
28 views

DQN cannot learn or converge

I have implemented a DQN using keras. The task is to collect the circles and avoid the red circle and crosses. The associated rewards are +5, -5 and 0 otherwise. if the agent go out of the board, the ...
0
votes
1answer
168 views

What is the relationship between MDP and RL?

What is the relationship between Markov Decision Processes and Reinforcement Learning? Could we say RL and DP are two types of MDP?
1
vote
0answers
27 views

Reinforcement learning - How to deal with varying number of actions which do number approximation

I am a new to Reinforcement learning, but I am trying to use RL in this task: Given a function definition in written e.g. in C with 1 to 10s of input arguments (only numerical ones - integer, float, ...
2
votes
1answer
25 views

What are features in the context of reinforcement learning?

In machine learning, "feature" is a synonym for explanatory variables. I know what a feature is. However, in the specific case of RL, it's not clear to me what features are. What are "features" in the ...
1
vote
1answer
193 views

what is difference between the DDQN and DQN?

I think I did not understand what is the difference between DQN and DDQN in implementation. I understand that we change the traget network during the running of DDQN but I do not understand how it is ...
1
vote
1answer
36 views

Boundaries of Reinforcement Learning

I finally developed a Game Bot that learns how to play the videogame Snake with Deep Q-Learning. I tried with different neural networks and hyper-parameters, and I found a working set-up, for a ...
4
votes
1answer
69 views

RL's policy gradient (REINFORCE) pipeline clarification

I try to build a policy gradient RL machine, and let's look at the REINFORCE's equation for updating the model parameters by taking a gradient to make the ascent (I apologize if notation is slightly ...
1
vote
1answer
52 views

Should reinforcement learning always assume (PO)MDP?

I recently just started learning reinforcement learning and learned that reinforcement learning algorithms work under the assumption of MDP or POMDP. However as I read A3C and recent vision based deep ...
1
vote
0answers
77 views

Policy gradient: why does this converge with Adam and not SGD?

I am looking into policy gradient methods. I stumbled into this implementation: https://gist.github.com/calclavia/cfcd41ad4e47d7b9b6ab8af15410747a It uses a Nesterov Adam optimizer. If I run it, it ...
2
votes
1answer
30 views

Reinforcement learning: Discounting rewards in the REINFORCE algorithm

I am looking into the REINFORCE algorithm for reinforcement learning. I am having trouble understanding how rewards should be computed. The algorithm from Sutton & Barto: What does G, 'return ...
0
votes
1answer
33 views

Can RL learn to mimic a hash function?

Theoretically, is this possible: When you have a given input string, there are a set of permutations and bit operations to do. The problem is, when you choose an approach like reinforcement learning, ...
1
vote
2answers
38 views

Equations in “Intoduction to RL”: What is the meaning and difference between E, and E with subscript?

This question is from An introduction to RL, page 78. In the formula below the page, both $\mathbb{E}$ and $\mathbb{E_\pi}$ are mentioned. Could you help me understand the difference between ...
1
vote
1answer
39 views

Dueling DQN - why should we decompose and then combine them back into?

could anyone who can help me if we decompose them and combine back them into a single Q, what the network can learn? from my perspective,the V means the total reward when the agent follow the current ...
0
votes
1answer
56 views

Policy Gradient Methods - ScoreFunction & Log(policy)

In Policy Gradient Methods, Lecture 7 (34:15), David describes a Score Function as being the Gradient of the Log of the policy Question: If we have a Neural ...
0
votes
0answers
41 views

Regularization and non determinism

I have a Deep Q-network which tries to learn how to play the game. This game however is non deterministic in a way that one action in a same state does not always produce the same outcome. One of the ...
3
votes
1answer
159 views

Reinforcement learning: decreasing loss without increasing reward

I'm trying to solve OpenAI Gym's LunarLander-v2. I'm using the Deep Q-Learning algorithm. I have tried various hyperparameters, but I can't get a good score. Generally the loss decreases over many ...
2
votes
1answer
47 views

Optimization motion planning by using Bellman equation

From the Montana article "Kinematics of Contact and Grasp", if I have a ball roll on the plane without sliding, the motion equation is described below: \begin{equation*} \begin{bmatrix} \dot{u}_{2} \\ ...
1
vote
1answer
357 views

RL Advantage function why A = Q-V instead of A=V-Q?

In RL Course by David Silver - Lecture 7: Policy Gradient Methods, David explains what an Advantage function is, and how it's the difference between Q(s,a) and the V(s) Preliminary, from this post: ...
2
votes
1answer
33 views

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )? I want to replace the DNN section of Qlearning with a RF or SVR but the problem is that there is no clear ...
1
vote
1answer
26 views

Is the neural network in DQN used to learn like a supervised model?

Is the neural network in DQN used to learn like a supervised model?
3
votes
1answer
54 views

How to use a different model to deep neural network with reinforcement learning based on DQN?

Is it possible to implement a reinforcement learning algorithm without using a deep neural network (DNN) as used in deep reinforcement learning e.g. Deep Q-Network (DQN)? How can I replace the DNN in ...
1
vote
1answer
27 views

MDP - RL, Multiple rewards for the same state possible?

This question is from An introduction to RL Pages 48 and 49. This question may also be related to below question, although I am not sure: Cannot see what the "notation abuse" is, mentioned ...