Questions tagged [actor-critic]
The actor-critic tag has no usage guidance.
20
questions
0
votes
0answers
8 views
saving a model during training of an RL agent
I am training an RL agent using PPO2 algorithm. Iam using stable-baselines library. During the training process, my rewards are slowly increasing and stabilizing, but are falling down suddenly. I ...
0
votes
0answers
18 views
Sudden decline in cumulative reward of an reinforcement learning training agent
Hi I am training an RL agent using PPO algorithm of stable baselines library. I have integrated my training with tensorboard to monitor the training process. Over a period of time my training rewards ...
0
votes
1answer
23 views
Different results every time I train a reinforcement learning agent
I am training an RL agent for a control problem using PPO algorithm. I am using stable-baselines library for it.
The objective of an agent is to maintain a temperature of 24 deg in a zone and it ...
1
vote
1answer
26 views
Evaluating a trained Reinforcement Learning Agent?
I am new to reinforcement learning agent training. I have read about PPO algorithm and used stable baselines library to train an agent using PPO. So my question here is how do I evaluate a trained RL ...
0
votes
0answers
10 views
RL library for implementing PPO algorithm
I have recently studied about PPO algorithm. I want to train an RL agent using PPO. However I am finding it difficult to implement. So I am planning to use a library which will help me in the ...
1
vote
2answers
91 views
Agent always takes a same action in DQN - Reinforcement Learning
I have trained an RL agent using DQN algorithm. After 20000 episodes my rewards are converged. Now when I test this agent, the agent is always taking the same action , irrespective of state. I find ...
0
votes
1answer
22 views
Reward is converging but actions taken by trained agent are illogical in reinforcement learning
I am training a reinforcement learning agent using DQN. My state space has 6 variables and the agent can one action which is discretized into 500 actions
My reward structure looks like
...
0
votes
0answers
33 views
Having Discrete and Continuous actions in Reinforcement Learning
I would like to design an agent that takes three actions in an environment. Two of which are continuous while the third is discrete.
Is there an algorithm in RL that can accomplish this?
0
votes
0answers
5 views
Actor-critic: Should actor and critic networks have the same size?
I'm using an actor-critic RL approach (DDPG to be specific) and am wondering if there is some rule of thumb that says whether or not the actor NN and critic NN should have the same size, ie, number of ...
1
vote
1answer
13 views
Why can't Policy Gradient Algorithm be seen as an Actor-Critic Method?
During the equation deducing in policy gradient algorithm(e.g., REINFORCE), we are actually using an expectancy of total reward, which we try to maximize.
$$\overline{R_\theta}=E_{\tau\sim\pi_\theta}[...
0
votes
1answer
108 views
A2C Continuous for Pendulum-v0 working implementation, negation for loss and entropy calculation
very good implementation of A2C continuous for Pendulum-v0
Code has snippet to stop execution when mean of last 10 or 20 is higher than -20 but the results look like:
...
0
votes
0answers
99 views
Implementation of actor-critic model for MountainCar
I'm trying to build a model for the Mountain Car game, following this Actor-Critic code: https://github.com/nikhilbarhate99/Actor-Critic
(However, in this case, it's discrete action space, while it's ...
0
votes
0answers
26 views
Speeding up actor critic training
I'm simulating a very simple system, recommendation system, and I am running an actor-critic model to predict what item I should recommend next. The agent is learning and is doing just fine. However, ...
0
votes
1answer
108 views
multipying negated gradients by actions for the loss in actor nn of DDPG
In this Udacity project code that I have been combing through line by line to understand the implementation, I have stumbled on a part in class Actor where this ...
1
vote
0answers
25 views
Reinforcement learning - generating a matrix of continuous values with varying size for test data generation
Currently, I am using RL A3C algorithm for test data generation, where for a set of 30 functions written in C (mostly basic algorithms like Prime number checks, triangle validity, etc.) I try to ...
0
votes
2answers
30 views
Proof subtracting baseline doesn't influence gradient can be used to show no gradient exist at all?
I am using David Silver's course in RL to help me write my thesis. However, I am baffled by the proof given in lecture 7 slide 29:
slideshow
\begin{align}
\mathbb{E}_{\pi_\theta}[\nabla_\theta \log_\...
1
vote
1answer
54 views
Stability of value function approximation in policy gradients
In DQNs, function approximation of the Q-values is unstable for correlated updates. In policy gradients with a baseline, will the value function of the policy not be plagued by the same correlated ...
3
votes
1answer
141 views
Time horizon T in policy gradients (actor-critic)
I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture.
At the bottom of that slide, the gradient of the expected sum of rewards ...
2
votes
2answers
533 views
A3C - Turning action probabilities into intensities
I'm experimenting with using an A3C network to learn to play old Atari video games. My network outputs a set of probabilities for each possible action (e.g. left, right, shoot), and I use this ...
3
votes
1answer
522 views
How to design two different neural nets for actor and critic RL?
In order to have an actor critic RL model there are two things to be satisfied .
Value approximation function should converge to a local minimum
$$\sum_s d^{\pi}(s) \sum_a \pi(s,a)[Q^{\pi}(s,a) - ...
