Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

2
votes
1answer
17 views

Pytorch: How to create an update rule the doesn't come from derivatives?

I want to implement the following algorithm, taken from this book, section 13.6: Here, the neural networks' outputs are $V(S, w)$ and $\pi(A|S,\theta)$, parameterized by $w$ and $\theta$ respectively....
1
vote
0answers
15 views

Hindsight experience replay: strategy for sampling goals

The authors of Hindsight Experience Replay list out several strategies for sampling a set of additional goals $G$ in Section 4.5: final - corresponds with final state of environment, future — replay ...
0
votes
0answers
13 views

Can we do convolutions on binary mask inputs?

I am training a vehicle trajectory prediction algorithm using Deep MaxEnt Inverse Reinforcement Learning (https://arxiv.org/abs/1507.04888). My intention is to have as input to this algorithm a top-...
3
votes
1answer
31 views

Is there a rule of thumb when designing neural network in deep reinforcement learning?

In deep learning, we can assess model's performance with loss function value and improve model's performance with K-fold cross-validation and so on. But how can we design and tune neural network used ...
0
votes
1answer
13 views

Proof subtracting baseline doesn't influence gradient can be used to show no gradient exist at all?

I am using David Silver's course in RL to help me write my thesis. However, I am baffled by the proof given in lecture 7 slide 29: slideshow \begin{align} \mathbb{E}_{\pi_\theta}[\nabla_\theta \log_\...
0
votes
0answers
8 views

What information should be cached in experience replay for actor-critic? [migrated]

In experience replay, we save a buffer of transactions $e_t$: $e_t = (s_t,a_t,r_t,s_{t+1})$ The equations for calculating the loss in actor critic are: Actor_Loss (parameterized by $\theta$): $...
0
votes
0answers
17 views

Q-Learning where some actions are more difficult to predict than others

I'm trying to train a deep Q network to optimize play in a game. For simplicity, let's say my game only has two actions, A and B. The reward distribution for A is somewhat uniform in the range -1 to ...
0
votes
2answers
20 views

Gym action space for board game with reward function

Im trying to design an openai gym environment that plays a quite simple board game where each player has 16 pieces that are exactly the same in regard to how they can move. The board is 10x10 and ...
0
votes
0answers
8 views

Is DQN a SMART (Semi Markov Average Reward Technique)?

Is DQN a SMART (Semi Markov Average Reward Technique) algorithm?
0
votes
0answers
17 views

About applying time series forecasting to problems better suited for reinforcement learning, like toy example “Jack's car rental”

"Jack's car rental" is an example of a reinforcement learning problem, proposed in the Sutton & Barto book, in which the goal is to optimize the daily distribution of cars in two locations of the ...
0
votes
1answer
14 views

Why is “next state” kept in RL experience replay?

Following this explanation on what is experience replay (and others), I noticed an experience element is defined as $e_t = (s_t,a_t,r_t,s_{t+1})$ My question is, why do we need the ...
1
vote
0answers
17 views

What is the meaning of the Variant Q-learning and To what INPUT and OUTPUT refer? in Abstract of DeepMind DQN paper 2013

-INPUT and OUTPUT OF ATARI DQN: In the abstract paragraph of the DQN work by DeepMind https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf it has written: " We present the first deep learning model to ...
2
votes
1answer
93 views

What is the difference between dynamic programming and Q-learning?

What is the difference between the DP-based algorithm and Q-learning?
0
votes
0answers
7 views

Proxy task for CNN architecture search

I would like to explore possible model's architectures to improve the object detection accuracy. The hyper-parameters I would like to optimize are the number of filters for each convolution layer and ...
1
vote
0answers
17 views

In DQN, what is the real reason we don't backpropagate through the target network?

Actually, if we are to backpropagate through the target network, there is no use for the target network anymore. Let's say we don't use any target network. We hence enforce only the "temporal ...
0
votes
0answers
16 views

How apply Reinforcement Learning in the following case?

Suppose that I have to move from point A to B and I have to choose among 3 different paths. But we don't know the traffic in each path, so what is the training rule to use to learn the best behaviour? ...
0
votes
1answer
26 views

RL - Weighthing negative rewards [closed]

Let's consider that I give an agent a reward of -1 (minimum reward) every time it performs an action which leads to the premature end of the episode (i.e., the agent dies). Besides, I also give a ...
0
votes
1answer
79 views

Reinforcement learning for continuous state and action space

Problem My goal is to apply Reinforcement Learning to predict the next state of an object under a known force in a 3D environment (the approach would be reduced to supervised learning, off-line ...
0
votes
1answer
20 views

How to represent an image as state in a Q-table

I'm trying to do Q-learning with the Atari games using the gym python's package. I want to use the image as the state of my algorithm, but I came up with a doubt: ...
0
votes
0answers
23 views

OpenAi CarRacing-v0 true speed?

True speed is defined in line 445 of https://github.com/openai/gym/blob/master/gym/envs/box2d/car_racing.py as ...
0
votes
0answers
23 views

How to interpret/understand Q-table in FrozenLake

I am again to invite attention to the example code for FrozenLake at this question link. The actual code is from here. The full code is again reproduced below. I am unable to interpret the final Q-...
0
votes
0answers
26 views

Which one to maximise? Q-value or V-value?

I just start learning reinforcement learning and studying Markov Decision Process (MDP). The state-value function (V-value, $v_{\pi}(s)$) and action-value function (Q-value, $q_{\pi}(s, a)$) are \...
0
votes
1answer
18 views

How does Q-Learning deal with mixed strategies?

I'm trying to understand how Q-learning deals with games where the optimal policy is a mixed strategy. The Bellman equation says that you should choose $max_a(Q(s,a))$ but this implies a single unique ...
0
votes
1answer
30 views

Why not use max(returns) instead of average(returns) in off-policy Monte Carlo control?

As I understood it, in reinforcement learning off-policy Monte Carlo control, when estimating the state-action value function Q(s, a), it is estimated as an weighted average of observed returns. ...
1
vote
1answer
30 views

Can Reinforcement learning be applied in image classification?

So my question is can Reinforcement learning be applied in image classification?
0
votes
0answers
11 views

formalizing the machine learning model training by incorporating the human involvement [on hold]

In some learning system designs, sometimes we incorporate a user-interface that enable the users or domain experts to manually change the models by some operations such as, change the rules or some ...
1
vote
1answer
18 views

In Reinforcement Learning can I randomly assign next_states from the state space to my agent while creating transition set?

In Reinforcement Learning, while creating transition samples (state, action, next_state, reward), where: Agent: The learning agent Environment: The trainer The environment gives two feedback to the ...
1
vote
1answer
53 views

Reinforcement learning: negative reward (punish) illegal actions?

If you train an agent using reinforcement learning (with Q-function in this case), should you give a negative reward (punish) if the agent proposes illegal actions for the presented state? I guess ...
0
votes
1answer
26 views

Policy gradient - and auto-differentiation (Pytorch/Tensorflow)

In policy gradient, we have something like this: Is my understanding correct that if I apply log cross-entropy on the last layer, the gradient will be automatically calculated as per formula above?
1
vote
1answer
26 views

Does policy optimization learn policies to make better actions with higher probability? [closed]

When I talk about policy optimization, it is referred to the following picture, and it is linked to DFO/Evolution plus Policy Gradients. I would like to know is it correct to say: Policy ...
0
votes
1answer
12 views

In first visit monte carlo are we assuming the environment is the same over episodes?

Watching this video (11:30) that presents the simplest algorithm for reinforcement learning: Monte Carlo Policy Evaluation, which says in general: The first time a sate is visited: increment N(s): N(...
0
votes
0answers
19 views

Natural actor-critic with Q function approximation

Referring to some useful readings, for example, this one: lectures on reinforcement learning, I try to program a natural gradient actor-critic, following this nice property: $$\nabla_\theta J(\theta) ...
1
vote
1answer
46 views

Reinforcement Learning - How are these state values in MRP calculated?

This is a question from the book an Introduction to RL, page 125, example 6.2. The example compares the prediction abilities of TD(0) and constant $ \alpha $ MC when applied to the below Markov ...
3
votes
1answer
168 views

How does Implicit Quantile-Regression Network (IQN) differ from QR-DQN?

As a newbie, for several months I browsed the internet hoping to find a user-friendly explanation of the Implicit Quantile Regression Network (IQN). But, it seems there is none at all. How does IQN ...
1
vote
1answer
32 views

TD Learning formula

This is something I cannot get my head around and initially I thought is a typo but it is not. Essentially in TD learning, we are trying to learn the Value Function. A value function tells me how ...
0
votes
1answer
228 views

IndexError: index 804 is out of bounds for axis 0 with size 800

i installed a self driving car project from superdatascience site , when i open the map using terminal after a while the map window close up or it closes directly after i maximize the map window and ...
0
votes
1answer
28 views

Do we need to use off-policy methods for policy shaping?

Let's say that there is a reinforcement learning task and an agent in a environment. I want a human teacher to manually modify the policy of the agent (policy shaping) to speed up the learning of the ...
1
vote
1answer
32 views

Will reinforcement learning work if states wont get repeated again?

I am working on a information retrieval model where the user enters a query and the model has to retrieve 3 most relevant FAQ pairs.I am collecting implicit feedback in terms of page clicks etc.What I ...
1
vote
0answers
38 views

Dueling DQN - Calculation of Q-value

I'm trying to implement a Double Dueling DQN on LunarLander and I'm facing an issue as my model is not learning so I'm trying to debug the graph and this leads me to a question regarding the ...
1
vote
1answer
27 views

objective in policy gradient equation?

I don't understand how this was deduced from first equation to second expectation. Is it from conditional probability theory? I checked but still can't understand. From wikipedia, the expectation of a ...
1
vote
1answer
27 views

Stability of value function approximation in policy gradients

In DQNs, function approximation of the Q-values is unstable for correlated updates. In policy gradients with a baseline, will the value function of the policy not be plagued by the same correlated ...
0
votes
0answers
20 views

Resampling to get equal predictive power per observation

This is probably a thing I am just not searching for correctly, but essentially my idea is this: given some machine learning classification $C$ based on an input dataset $D$, certain observations in $...
0
votes
1answer
45 views

What is “GOAL” in terms of Reinforcement Learning specified in these papers?

I have a question regarding Reinforcement Learning. I've been reading the Horde and the UVFA paper extensively. Take the Horde paper, there is this GVF, General Value Function Approximators which ...
2
votes
1answer
51 views

Is RL applicable to environments that are totally RANDOM?

I have a fundamental question on the applicability of reinforcement learning (RL) on a problem we are trying to solve. We are trying to use RL for inventory management - where the demand is entirely ...
1
vote
0answers
22 views

Deep RL: Visualizing/Analyzing the gradient

I am testing different RL methods, and I know e.g that policy gradient method is supposed to have a high variance gradient which can cause trouble. I want to run a few different Deep RL algorithms, ...
1
vote
1answer
46 views

What is the immediate reward in value iteration?

Suppose you're given an MDP where rewards are attributed for reaching a state, independently of the action. Then when doing value iteration: $$ V_{i+1} = \max_a \sum_{s'} P_a(s,s') (R_a(s,s') + \...
2
votes
1answer
65 views

Reinforcement learning: easily learnable state representation

I have created a simple OpenAI Gym environment, which consists of: A continuous 2D world with x and y in range [0.0, 1.0] A rabbit which slowly moves randomly in the world with a constant speed A '...
0
votes
0answers
22 views

have deep reinforcement learning algorithms the unified behavior? [on hold]

Could we say the behavior of different deep reinforcement learning algorithms is very similar for MDP and POMDP? Could we say DRL algorithms present a unified approach for finite-horizon, infinite ...
2
votes
1answer
58 views

What are the differences between Reinforcement Learning (RL) and Supervised Learning?

What is the difference between Reinforcement Learning (RL) and Supervised Learning? Does RL hava more difficulty in finding a stable solution? Does Q-learning have more difficulty in finding a ...
1
vote
0answers
50 views

DQN cannot learn or converge

I have implemented a DQN using keras. The task is to collect the circles and avoid the red circle and crosses. The associated rewards are +5, -5 and 0 otherwise. if the agent go out of the board, the ...