Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Filter by
Sorted by
Tagged with
2
votes
0answers
34 views

Policy Gradient methods not converging to useful mean values

I am getting familiar with Policy Gradient methods, specifically Advantage Actor Critic (A2C). My target problem use clipped continuous state and action spaces and I have therefore been training my ...
2
votes
1answer
128 views

Temporal difference learning with a neural network

Suppose I would like to train a value network $v$ via TD(0). So my TD target for a time step $t$ equals: $$R_{t+1} + \gamma v(s_{t+1})$$ If I understand correctly, I just need to use mean squared ...
0
votes
1answer
19 views

Reinforcement Learning control with known dynamic equation

I know there is model-based reinforcement learning. But all the approaches assume an MDP. If I want to do a feedback control of a system (i. e. control an inverted pendulum) it's quite easy to find ...
1
vote
1answer
37 views

Expected value in Bellman equation

I am reading "reinforcement learning - An introduction" by Sutton and Barto. At pag. 59, there is the Bellman equation for the state-value function $\begin{array}{ll} v_{\pi}(s) &= \mathbb{E}_{...
0
votes
0answers
21 views

Can dynamic pricing be modeled as Self Learning (Tabula rasa) Reinforcement Learning problem?

Mostly with respective Dynamic pricing problem, a large set of data is required to tutor or as guidance for reinforcement learning agents to avoid the cold start problems. Is there a way to define an ...
1
vote
2answers
122 views

Alternative approach for Q-Learning

I have a question related to an alterative Q-Learning approach. I'd like to know if this already exists and I am not aware of it, or it doesn't exist because there are theoretical problems behind it. ...
1
vote
0answers
23 views

DQNs for huge or continuous state spaces

Have there been occasions where DQN failed to deal with huge state spaces? Can you point out a research paper regarding it?
2
votes
1answer
87 views

Dueling Network gradient with respect to Advantage stream

Looking at Dueling DQN: $Q = V + A - mean(A)$ For simplicity, let's assume we are working with 4 neurons. Recall that Value stream only has 1 neuron $(v_0)$ Re-writing the above equation, we get: $...
0
votes
0answers
103 views

Q learning advantages

what is the advantages and disadvantages of using Q function in reinforcement learning comparing to other method such as policy gradient
7
votes
0answers
137 views

Formal proof of vanilla policy gradient convergence

So I stumbled upon this question, where the author asks for a proof of vanilla policy gradient procedures. The answer provided points to some literature, but the formal proof is nowhere to be included....
0
votes
2answers
42 views

RL Invertible Value Function approach - why it prevents rewards from exploding?

Authors of "Recurrent Experience Replay in Destributed RL", page 3, use the function $h$ to prevent rewards from exploding: $$h(x) = \operatorname{sign}(x)(\sqrt{|x|+1} -1) + \epsilon x$$ where $\...
0
votes
0answers
60 views

How are Q-value and state value different in reinforcement learning

I have gone through this answer regarding the difference between Q-value and state value. My specific question is: If Q-value calculates immediate reward after taking a particular action and then ...
0
votes
2answers
37 views

Does an RL agent learn during exploitation?

I have started with RL and have some doubts regarding it. Does an RL agent learn during exploitation, or does it only learn during exploration? Is it possible to train a model only using exploitation ...
0
votes
0answers
18 views

inverted pendulum like reinforcement learning

i want to use reinforcement learning to control a car with two actuated motors (servos). The motor are attached to one axis and on the axis is put a platform. The setup of the two actuated wheels is ...
3
votes
1answer
39 views

About the time differences in the Bellman equation

I am trying to grasp fundamental mathematics behind the Reinforcement Learning and so far I have unterstood how the Value Iteration and Policy algorithms do converge (contractions, etc.) I have still ...
0
votes
0answers
32 views

How to optimise a boolean expression

I am working on an optimization problem involving Boolean expressions and wanted some help as I have very little knowledge about the topic. The problem statement is as follows: There are a set of ...
1
vote
1answer
251 views

Difference between Dueling DQN and Double DQN?

I have read some articles, but still can not figure out the difference between the Dueling DQN and Double DQN? What exactly is the difference between them? Also, Does Dueling DQN need to be built on ...
1
vote
0answers
64 views

CartPole v1 - Simple backprop with 1 hidden layer

I'm trying to solve the CartPole-v1 problem from OpenAI by using backprop on a one-layer neural network - while updating the model at every time step using State action values (Q(s,a)). I'm unable to ...
0
votes
0answers
93 views

How to train DDQN model in reinforcement learning?

I was reading through some RL code but couldn't understand one small bit. here is the code: ...
0
votes
0answers
56 views

Attempting to implement Linear Regression SGD Sarsa for Reinforcement Learning

UPDATE: The problem seemed to be simply due to setting the reward too high (at 50). Lowering alpha, the discount and most importanty the reward allowed convergence. I'm genuinly confused why a large ...
1
vote
0answers
34 views

Q-value estimate in neural episodic control

Disclaimer: I'm not that familiar with reinforcement learning, so I might lack some basic knowledge on that topic. I was reading the neural episodic control (NEC) paper by Pritzel et al., 2017 and I'...
2
votes
0answers
71 views

Optimal implementation of vanilla DQN loss in Keras

I've implemented vanilla DQN for continuous/non-images (no CNN) states in keras. But, I'm not sure if my implementation of the loss computation is optimal. For reminder the loss is defined as : $loss=...
2
votes
0answers
20 views

Best learning automata reinforcement schema for solving grid world,help me [closed]

I have a gridworld puzzel , an agent and Target , I want find best path for reaching Target by agent. gridworld example(with S as start point, G as goal point and black cells as cliffs):
2
votes
0answers
73 views

Reinforcement Learning using PPO2 in openai gym retro, mario not learning the clear the easy episode

I am training mario game in retro using ppo2 baselines for some time. I have tried level3 and level1 too. But even after full training when I play using saved checkpoints, the mario is not able to ...
0
votes
0answers
22 views

Fetch (multi joint robot) DQN training: How to do action selection?

I am implementing a DQN using a similar environment to OpenAI fetch envs. I am trying to convert them to pybullet implementations. When training a DQN for a multi-joint robot like the Fetch, ...
0
votes
0answers
315 views

RL ppo alrorithm: understanding value loss and entropy plot

I'm implementing a computer vision program using PPO alrorithm mostly based on this work Both the critic loss and the actor loss decrease in the first serveal hundred episodes and keep near 0 later(...
0
votes
0answers
17 views

Reinforcement Learning - Q Learning - Number of Steps to Decrease?

I have an implementation of the Q-Learning algorithm intended to solve the racetrack problem. I have noticed that the initial amount of steps needed to solve the problem is somewhere between 3000-...
0
votes
0answers
16 views

how can I find tuned X feature values and Minimized Y value?

I am starting to work on a smart-factory project. Now we would like to achieve as below: 1: Minimizing cost (y value) 2: getting the best tuned (optimized) value of X features. (e.g., now ...
0
votes
0answers
19 views

Run Reinforcement Learning using PPO saved model

I have gone through this good article on reinforcement learning, Reinforcement Learning - contra game, In this author has trained RL engine using PPO2 policy and saved the model in ppo_save folder. ...
0
votes
0answers
129 views

How to evaluate reinforcement learning model?

I am relatively new to reinforcement learning and have been experiencing with a reinforcement learning model to make decisions based on human activities (dynamic environment). Appreciate if someone ...
1
vote
0answers
48 views

Intuition behind the loss function in Deep Q learning?

I'm currently following a tutorial but I got stuck at the deep Q learning model. According to my understanding of neural networks they predict an approximate function for the inputs given with the ...
2
votes
1answer
23 views

If the set of all possible states changes each time, how can Q-learning “learn” anything?

I found this resource that explains q-learning with a very simple example. Make it a 2D problem, a rectangle instead of a line, and it's still simple. The only difference is that now there are 2 more ...
2
votes
1answer
31 views

Representing state in Q-Learning

I have a fairly simple game in which I wish to use Q-learning to train an agent, but I have some questions regarding state representation. I'm new to RL so bare with me: If you have a game where you ...
0
votes
1answer
113 views

How does DQN solve Open AI Cartpole - v0?

Context I am confused about how a DQN is supposed to solve the cart pole problem since the rewards are so dense. I have been using pytorch example. I am aware of some solutions, but I have issue with ...
0
votes
1answer
25 views

Is it possible to measure the object using deep learning

Is there a way we can measure the length, width and the depth of an object in the picture using deep learning?
0
votes
0answers
53 views

Why is the reward fluctuating for Double Q-Learning?

I am trying to implement Double Q-Learning using neural networks from the Keras library. When I first tried Simple DQN, the graph of the reward was fluctuating a lot so, I implemented a Double DQN. ...
1
vote
1answer
1k views

PPO, A2C for continuous action spaces, math and code

Edit: Question has been edited to better reflect what I learned after asking the original question. I implemented the clipped objective PPO-clip as explained here: https://spinningup.openai.com/en/...
1
vote
0answers
28 views

Dynamic pricing models when there are a lot of products with low demand

For products with high demand and much stock, there are several algorithms that work for dynamic pricing. However, when selling many different products in the same category and demand for each one is ...
1
vote
1answer
287 views

What will be the policy if the state space is continuous in Reinforcement learning

I have started recently with reinforcement learning. I have few doubts regarding the policy of an agent when it comes to continuous space. From my understanding, policy tells the agent which action ...
4
votes
1answer
272 views

Large action space for deep reinforcement learning

I know that in normal Deep Reinforcement Learning(DRL) scenario, we learn a deep neural network to map current states to Q values. The number of the Q values (# of outputs of the neural network) is ...
2
votes
1answer
128 views

Looking for an RL solution for sequence generation

I am looking for a proper reinforcement learning solution for the following problem: Suppose I have a pool of candidate functions f \in Pool(it's like ...
0
votes
0answers
72 views

tensorflow eager execution outputs only same values

I'm trying to convert my tensorflow code to tensorflow eager. The problem is the forward pass predicts only the same actions for different input values in eager mode. The normal tensorflow code with ...
0
votes
1answer
268 views

Q-Learning experience replay: how to feed the neural network?

I'm trying to replicate the DQN Atari experiment. Actually my DQN isn't performing well; checking another one's codes, I saw something about experience replay which I don't understand. First, when you ...
0
votes
1answer
122 views

Policy gradient/REINFORCE algorithm with RNN: why does this converge with SGM but not Adam?

I am working on training RNN model on caption generation with REINFORCE algorithm. I adopt self-critic strategy (see paper Self-critical Sequence Training for Image Captioning) to reduce the variance. ...
0
votes
0answers
22 views

Different algorithms categorized in reinforcement learning

(Originally asked at cross validated forum: https://stats.stackexchange.com/questions/401615/different-algorithms-categorized-in-reinforcement-learning) For some time I am going through reinforcement ...
1
vote
1answer
84 views

Use deep reinforcement learning with recursive actions?

Can we use recursive actions in deep reinforcement learning ? If yes how ? For example, in a reassignment problem, for a task i the neural networks returns a node m, and then there will be some ...
1
vote
0answers
78 views

REINFORCE algorithm with discounted rewards – where does gamma^t in the update come from?

I'm looking at Sutton & Barto's rendition of the REINFORCE algorithm (from their book here, pg. 328). I can't quite understand why there is $\gamma^t$ on the last line. They say: [..] in the ...
0
votes
0answers
130 views

Implementation of actor-critic model for MountainCar

I'm trying to build a model for the Mountain Car game, following this Actor-Critic code: https://github.com/nikhilbarhate99/Actor-Critic (However, in this case, it's discrete action space, while it's ...
2
votes
1answer
343 views

DQN fails to find optimal policy

Based on DeepMind publication, I've recreated the environment and I am trying to make the DQN find and converge to an optimal policy. The task of an agent is to learn how to sustainably collect apples ...
0
votes
0answers
29 views

Speeding up actor critic training

I'm simulating a very simple system, recommendation system, and I am running an actor-critic model to predict what item I should recommend next. The agent is learning and is doing just fine. However, ...