Skip to main content

Questions tagged [policy-gradients]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
62 views

Deep Q-Learning: How are network parameters updated, and why consider episodes in the first place?

I'm trying to wrap my head around the implementation of deep $Q$-learning, and why we even consider episodes in the first place. The usual set-up is that we initialize some starting state $s_0$, then ...
infinitylord's user avatar
0 votes
0 answers
193 views

Reinforcement learning with PPO - tips and tricks

I am trying to use PPO where the agent has to maneuver around an obstacle towards the target while respecting the spatial boundaries. While the agent learns to respect spatial boundaries it never ...
SathukaBootham's user avatar
0 votes
0 answers
14 views

Policy Gradient training log-derivative un-normalized vs normalized objective

I am implementing a policy gradient training objective for optimizing ranking metrics in a learning-to-rank setting. For a given query $q$, a set of documents $D_q$ (retrieved from a first-stage ...
SHASHANK GUPTA's user avatar
0 votes
1 answer
28 views

The role of policy optimization in model-based RL

So I have a simulation $M_{sim}$ that approximates a nonlinear dynamic robotic model $M_{real}$ by solving a set of nonlinear differential equations. Given $M_{sim}$, I use an agent $A$ (namely, I use ...
Hadar's user avatar
  • 147
0 votes
1 answer
57 views

What does this value function of a parameterized policy mean? and is it related to TRPO in RL?

Iv been watching the RL lectures on youtube from Stanford. In episode 9 – policy gradients 2 the teacher Emma Brunskill says we are going to learn about how to make safer policy gradient steps by ...
Matthew Buchanan's user avatar
1 vote
1 answer
164 views

experience replay memory: saving the next state required when state does not depend on action?

so, I am using an agent with a state-action-policy and I am trying to understand the concept of experience replay memory (ERM). As far as I learned until now, the ERM is basically a buffer that stores ...
user101893's user avatar
0 votes
1 answer
26 views

Understanding derivation of gradient optimisation problem

I'm following a tutorial on youtube about reinforcement learning. They are going through the steps to understand policy gradient optimisation. In one of the steps he says ...
Funzo's user avatar
  • 113
1 vote
1 answer
712 views

Does GPU decreases training time for on-policy RL?

I was wondering whether using a GPU will be effective if I am using an on-policy (eg PPO) RL as the model? I.e, how can we use a GPU to decrease training time for an on-policy RL model? I recently ...
Wenuka's user avatar
  • 95
0 votes
1 answer
171 views

Which Algorithm did OpenAI used to create a hide and seek playing Agent?

I just saw this video on youtube: https://www.youtube.com/watch?v=kopoLzvh5jY&t=9s Which Algorithm did OpenAI used to create a hide and seek playing Agent? Was it Genetic Algorithm or Policy ...
learner's user avatar
3 votes
1 answer
51 views

Which Policy Gradient Method was used by Google's Deep Mind to teach AI to walk

I just saw this video on Youtube. Which Policy Gradient method was used to train the AI to walk? Was it DDPG or D4PG or what?
learner's user avatar
  • 33
1 vote
1 answer
29 views

Why is DDPG a Policy Gradient Method? [closed]

Why is DDPG a Policy Gradient Method even though it's actor does not output probability?
learner's user avatar
  • 21
0 votes
1 answer
40 views

How did the author got this final result from this Gaussian Distribution formula? [closed]

How did they got the final result?
learner's user avatar
  • 43
0 votes
1 answer
352 views

Which policy gradient method is used for continuous action spaces?

Which policy gradient method is used that deals with continuous action spaces?
learner's user avatar
  • 43
1 vote
1 answer
1k views

Reinforcement Learning - PPO: Why do so many implementations calculate the returns using the GAE? (Mathematical reason)

There are so many PPO implementations that use GAE and do the following: ...
Johannes's user avatar
0 votes
1 answer
30 views

How is a policy expressed? [closed]

In my work in behavioural cloning, I have been asked 'how is your policy expressed?' and I didn't know the answer to this. I was trying to create apply a behavioural cloning algorithm from the context ...
cgo's user avatar
  • 133
1 vote
1 answer
99 views

How is this score function estimator derived?

In this paper they have this equation, where they use the score function estimator, to estimate the gradient of an expectation. How did they derive this?
adam's user avatar
  • 13
1 vote
0 answers
138 views

A2C learning very slowly when I try to make it learn on batches as compared to making it learn on each step

I tried this on openai gym environment - LunarLander-v2. I wrote two algorithms with just one difference: Made it learn on each step. Made it learn at the end of each episode. There is a ...
starlord's user avatar
3 votes
1 answer
292 views

Policy Gradient not "learning"

I'm attempting to implement the policy gradient taken from the "Hands-On Machine Learning" book by Geron, which can be found here. The notebook uses Tensorflow and I'm attempting to do it with PyTorch....
Harpal's user avatar
  • 913
1 vote
0 answers
265 views

Entropy applied to policy gradient prevent our agent from being stuck in the local minimum?

In the information theory, the entropy is a measure of uncertainty in some system. Being applied to agent policy, entropy shows how much the agent is uncertain about which action to make. In math ...
jgauth's user avatar
  • 131
1 vote
2 answers
113 views

Reinforcement Learning : Why acting greedily with the optimal value function gives you the optimal policy?

The course of David Silver about Reinforcement Learning explains how you get the optimal policy from the optimal value function. It seems to be very simple, you just have to act greedily, by ...
tristan's user avatar
  • 11
6 votes
1 answer
138 views

Reinforcement Learning: Policy Gradient derivation question

I have been reading this excellent post: https://medium.com/@jonathan_hui/rl-policy-gradients-explained-9b13b688b146 and following the RL-videos by David Silver, and I did not get this thing: For $\...
Hadamard's user avatar
1 vote
1 answer
119 views

How to improve tensorflow 2.0 code for policy gradient?

I recreated some code I found online for solving the bandits problem using policy gradient. The example was in tensorflow 1.0 so I recreated it with tensorflow 2.0 using eager execution and gradient ...
Juan Acevedo's user avatar
3 votes
1 answer
304 views

Maximum Entropy Policy Gradient Derivation

I am reading through the paper on Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review by Sergey Levine. I am having a difficulty in understanding this part of the ...
Ricky Sanjaya's user avatar
3 votes
1 answer
158 views

reinforcement learning: Decompose a policy gradient

I am studying the policy gradient through the website: https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f Couldn't figure out how the first equation becomes the second ...
Edamame's user avatar
  • 2,755
2 votes
1 answer
6k views

Policy Gradient with continuous action space

How to apply reinforce/policy-gradient algorithms for continuous action space. I have learnt that one of the advantages of policy gradients is , it is applicable for continuous action space. One way I ...
chink's user avatar
  • 555
4 votes
2 answers
5k views

Agent always takes a same action in DQN - Reinforcement Learning

I have trained an RL agent using DQN algorithm. After 20000 episodes my rewards are converged. Now when I test this agent, the agent is always taking the same action , irrespective of state. I find ...
chink's user avatar
  • 555
1 vote
1 answer
119 views

Policy Gradient custom loss function not working

I was experimenting with my policy gradient reinforcement learning algorithm, and I was wondering if I could use a similar method to the supervised cross-entropy. So, instead of using existing labels, ...
Ross Myhovych's user avatar
0 votes
1 answer
93 views

Reward is converging but actions taken by trained agent are illogical in reinforcement learning

I am training a reinforcement learning agent using DQN. My state space has 6 variables and the agent can one action which is discretized into 500 actions My reward structure looks like ...
chink's user avatar
  • 555
1 vote
1 answer
215 views

How do the policy gradient's cost function and gradients work?

I am not a math expert but have a basic understanding of linear algebra, calculus and probability and I understand the math behind back propagation. Currently I am trying to learn about policy ...
Eka's user avatar
  • 301
1 vote
0 answers
472 views

Reducing the training time of an RL agent

I am trying to develop an rl agent using DQN algorithm.During training, the agent interacts with environment which is a simulated one.Each episode takes around 10 mins to run. This way if want my ...
chink's user avatar
  • 555
1 vote
1 answer
604 views

Policy gradient vs cost function

I was working with continuous system RL and obviously stumbled across this Policy Gradient. I want to know is this something like cost function for RL? It kinda gives that impression considering we ...
Sarvagya Gupta's user avatar
2 votes
2 answers
322 views

Guidelines to debug REINFORCE-type algorithms?

I implemented a self-critical policy gradient (as described here), for text summarization. However, after training, the results are not as high as expected (actually lower than without RL...). I'm ...
Astariul's user avatar
  • 1,004
1 vote
0 answers
32 views

Learning curve goes down after converge?

I trained an agent with policy gradient and the learning curve starts to decrease after it converges. I am wondering if this is overfitting or if this is due to some other issues?
beepretty's user avatar
  • 111
1 vote
1 answer
145 views

Why can't Policy Gradient Algorithm be seen as an Actor-Critic Method?

During the equation deducing in policy gradient algorithm(e.g., REINFORCE), we are actually using an expectancy of total reward, which we try to maximize. $$\overline{R_\theta}=E_{\tau\sim\pi_\theta}[...
KningTG's user avatar
  • 11
0 votes
0 answers
85 views

Using reinforce algorithm with per-action reward instead of per-trajectory reward

I've found some articles that talk about the reinforce algorithm / monte carlo method. The algorithm boils down to using this equation. The right summation over the trajectory is the reward for the ...
Thomas's user avatar
  • 101
2 votes
0 answers
359 views

Policy Gradient methods not converging to useful mean values

I am getting familiar with Policy Gradient methods, specifically Advantage Actor Critic (A2C). My target problem use clipped continuous state and action spaces and I have therefore been training my ...
The_Chicken_Lord's user avatar
23 votes
2 answers
623 views

Formal proof of vanilla policy gradient convergence

So I stumbled upon this question, where the author asks for a proof of vanilla policy gradient procedures. The answer provided points to some literature, but the formal proof is nowhere to be included....
Markus Peschl's user avatar
2 votes
1 answer
360 views

Policy gradient/REINFORCE algorithm with RNN: why does this converge with SGM but not Adam?

I am working on training RNN model on caption generation with REINFORCE algorithm. I adopt self-critic strategy (see paper Self-critical Sequence Training for Image Captioning) to reduce the variance. ...
Kechen's user avatar
  • 21
2 votes
0 answers
543 views

REINFORCE algorithm with discounted rewards – where does gamma^t in the update come from?

I'm looking at Sutton & Barto's rendition of the REINFORCE algorithm (from their book here, pg. 328). I can't quite understand why there is $\gamma^t$ on the last line. They say: [..] in the ...
Tuetschek's user avatar
  • 121
1 vote
1 answer
501 views

multipying negated gradients by actions for the loss in actor nn of DDPG

In this Udacity project code that I have been combing through line by line to understand the implementation, I have stumbled on a part in class Actor where this ...
mLstudent33's user avatar
1 vote
1 answer
90 views

In calculating policy gradients, wouldn't longer trajectories have more weight according to the policy gradient formula?

In Sergey Levine's lecture on policy gradients (berkeley deep rl course), he show that policy gradient can be evaluated according to the formula In this formula, wouldn't longer trajectories get more ...
liyuan's user avatar
  • 215
0 votes
1 answer
168 views

Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

I am looking for a little clarity on what the policy gradient theorem means. My confusion lies in the fact that the reward $R$ in reinforcement learning is non-differentiable in the policy parameters. ...
figs_and_nuts's user avatar
2 votes
0 answers
109 views

policy gradient loss [closed]

I am confused with the process for calculating loss. My code is below: ...
Kang_Kai's user avatar
2 votes
1 answer
348 views

Problem when cherry picking actions - Proximal Policy Optimization

I am using the implementation of PPO2 in stable-baselines (a fork of OpenAI's baselines) for a Reinforcement Learning problem. My observation space is $9x9x191$ and my action space is $144$. Given a ...
Max Fischer's user avatar
0 votes
2 answers
178 views

Proof subtracting baseline doesn't influence gradient can be used to show no gradient exist at all?

I am using David Silver's course in RL to help me write my thesis. However, I am baffled by the proof given in lecture 7 slide 29: slideshow \begin{align} \mathbb{E}_{\pi_\theta}[\nabla_\theta \log_\...
Alex Van de Kleut's user avatar
1 vote
1 answer
142 views

Why is "next state" kept in RL experience replay?

Following this explanation on what is experience replay (and others), I noticed an experience element is defined as $e_t = (s_t,a_t,r_t,s_{t+1})$ My question is, why do we need the ...
Gulzar's user avatar
  • 196
2 votes
1 answer
214 views

Policy gradient - and auto-differentiation (Pytorch/Tensorflow)

In policy gradient, we have something like this: Is my understanding correct that if I apply log cross-entropy on the last layer, the gradient will be automatically calculated as per formula above?
Jed's user avatar
  • 21
1 vote
1 answer
81 views

Does policy optimization learn policies to make better actions with higher probability? [closed]

When I talk about policy optimization, it is referred to the following picture, and it is linked to DFO/Evolution plus Policy Gradients. I would like to know is it correct to say: Policy ...
user10296606's user avatar
  • 1,834
1 vote
1 answer
156 views

Stability of value function approximation in policy gradients

In DQNs, function approximation of the Q-values is unstable for correlated updates. In policy gradients with a baseline, will the value function of the policy not be plagued by the same correlated ...
Johann's user avatar
  • 53
1 vote
0 answers
44 views

Deep RL: Visualizing/Analyzing the gradient

I am testing different RL methods, and I know e.g that policy gradient method is supposed to have a high variance gradient which can cause trouble. I want to run a few different Deep RL algorithms, ...
Carl Rynegardh's user avatar