Questions tagged [actor-critic]
The actor-critic tag has no usage guidance.
29
questions
0
votes
0
answers
28
views
Bellman Error for Value Function $V$
I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and ...
1
vote
0
answers
26
views
Actor-Critic one step TD update rule
In Sutton & Barto's book (Chapter $13$), it is stated that the update rule in REINFORCE could be reformated as
\begin{equation}
\begin{split}
\theta_{t+1}
&=\theta_t+\alpha\left(G_{t:t+1}-\hat{...
- 137
0
votes
1
answer
16
views
Why is the optimal output out of domain in A2C?
If each state has an optimal action, then the optimal actions distribution vector is a one-hot vector kind of like [0,0,1,0,0,0].
But with algorithms like A2C, we ...
- 1
1
vote
0
answers
68
views
What is the meaning about the $\alpha$ in TD3 algorithm
I am study the paper with TD3 algorithm.
I am curious about the meaning of $\alpha$ while the paper prove that overestimation will be happened in a critical situation.
The contents about mathematical ...
- 25
1
vote
0
answers
248
views
Soft actor-critic reinforcement learning for 100x100 maze environment
I am doing a project which requires a soft actor-critic reinforcement learning agent to learn how to reach a goal in a 100x100 maze environment as the one below:
The state space is discrete and only ...
0
votes
1
answer
309
views
Actor Network Target Value in A2C Reinforcement Learning
In DQN, we use;
$Target = r+\gamma v(s')$ equation to train (fit) our network. It is easy to understand since we use the $Target$ value as the dependent variable like we do in supervised learning. I....
- 53
1
vote
1
answer
1k
views
Reinforcement Learning - PPO: Why do so many implementations calculate the returns using the GAE? (Mathematical reason)
There are so many PPO implementations that use GAE and do the following:
...
- 11
1
vote
0
answers
135
views
A2C learning very slowly when I try to make it learn on batches as compared to making it learn on each step
I tried this on openai gym environment - LunarLander-v2.
I wrote two algorithms with just one difference:
Made it learn on each step.
Made it learn at the end of each episode.
There is a ...
- 11
1
vote
0
answers
235
views
Pytorch XLA to solve the spawn problems in a Colab Env
As reference only, here is my code
It seems that torch.multiprocessing.set_start_method("spawn") can't be used in an Colab Env. Only 'fork' is allowed.
I have ...
- 131
1
vote
0
answers
42
views
Action selection in actor-critic algorithm:
I have an action space that is just a list of values given by acts = [i for i in range(10, 100, 10)].
According to pytorch documentary, the loss is calculated as below. Could someone explain to me how ...
- 75
2
votes
0
answers
49
views
Actions taken by agentn/ agent performance not improving
Hi I am trying to develop an rl agent using PPO algorithm. My agent takes an action(CFM) to maintain a state variable called RAT in between 24 to 24.5. I am using PPO algorithm of stable-baselines ...
- 555
1
vote
0
answers
80
views
Rewards are converged but with a lot of variations
I am training a reinforcement learning agent on an episodic task of fixed episode length. I am tracking the training process by plotting the cumulative rewards over an episode. I am using tensorboard ...
- 555
2
votes
1
answer
463
views
Having a reward structure which gives high positive rewards compared to the negative rewards
I am training an RL agent using PPO algorithm for a control problem. The objective of the agent is to maintain temperature in a room. It is an episodic task with episode length of 9 hrs and step size(...
- 555
1
vote
1
answer
65
views
Formulation of a reward structure
I am new to reinforcement learning and experimenting with training of RL agents.
I have a doubt about reward formulation, from a given state if a agent takes a good action i give a positive reward, ...
- 555
1
vote
0
answers
291
views
Actor Critic Model implementation
I am going to work on a project which requires implementation of A2C model using Tensorflow 2.0. I am new in the Machine Learning field and also in Python. These are topics which I have covered ...
- 113
0
votes
1
answer
166
views
How to handle differences between training and deploying of an RL agent
Hi I am training an RL agent for a control problem. The objective of the agent is to maintain temperature in a zone. It is an episodic task with episode length of 10 hrs and actions being taken every ...
- 555
1
vote
1
answer
1k
views
Different results every time I train a reinforcement learning agent
I am training an RL agent for a control problem using PPO algorithm. I am using stable-baselines library for it.
The objective of an agent is to maintain a temperature of 24 deg in a zone and it ...
- 555
3
votes
1
answer
5k
views
Evaluating a trained Reinforcement Learning Agent?
I am new to reinforcement learning agent training. I have read about PPO algorithm and used stable baselines library to train an agent using PPO. So my question here is how do I evaluate a trained RL ...
- 555
4
votes
2
answers
4k
views
Agent always takes a same action in DQN - Reinforcement Learning
I have trained an RL agent using DQN algorithm. After 20000 episodes my rewards are converged. Now when I test this agent, the agent is always taking the same action , irrespective of state. I find ...
- 555
0
votes
1
answer
87
views
Reward is converging but actions taken by trained agent are illogical in reinforcement learning
I am training a reinforcement learning agent using DQN. My state space has 6 variables and the agent can one action which is discretized into 500 actions
My reward structure looks like
...
- 555
1
vote
1
answer
142
views
Why can't Policy Gradient Algorithm be seen as an Actor-Critic Method?
During the equation deducing in policy gradient algorithm(e.g., REINFORCE), we are actually using an expectancy of total reward, which we try to maximize.
$$\overline{R_\theta}=E_{\tau\sim\pi_\theta}[...
- 11
1
vote
1
answer
987
views
A2C Continuous for Pendulum-v0 working implementation, negation for loss and entropy calculation
very good implementation of A2C continuous for Pendulum-v0
Code has snippet to stop execution when mean of last 10 or 20 is higher than -20 but the results look like:
...
- 574
1
vote
1
answer
489
views
multipying negated gradients by actions for the loss in actor nn of DDPG
In this Udacity project code that I have been combing through line by line to understand the implementation, I have stumbled on a part in class Actor where this ...
- 574
1
vote
0
answers
84
views
Reinforcement learning - generating a matrix of continuous values with varying size for test data generation
Currently, I am using RL A3C algorithm for test data generation, where for a set of 30 functions written in C (mostly basic algorithms like Prime number checks, triangle validity, etc.) I try to ...
- 21
0
votes
2
answers
143
views
Proof subtracting baseline doesn't influence gradient can be used to show no gradient exist at all?
I am using David Silver's course in RL to help me write my thesis. However, I am baffled by the proof given in lecture 7 slide 29:
slideshow
\begin{align}
\mathbb{E}_{\pi_\theta}[\nabla_\theta \log_\...
1
vote
1
answer
148
views
Stability of value function approximation in policy gradients
In DQNs, function approximation of the Q-values is unstable for correlated updates. In policy gradients with a baseline, will the value function of the policy not be plagued by the same correlated ...
- 43
4
votes
1
answer
409
views
Time horizon T in policy gradients (actor-critic)
I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture.
At the bottom of that slide, the gradient of the expected sum of rewards ...
2
votes
2
answers
624
views
A3C - Turning action probabilities into intensities
I'm experimenting with using an A3C network to learn to play old Atari video games. My network outputs a set of probabilities for each possible action (e.g. left, right, shoot), and I use this ...
- 143
3
votes
1
answer
1k
views
How to design two different neural nets for actor and critic RL?
In order to have an actor critic RL model there are two things to be satisfied .
Value approximation function should converge to a local minimum
$$\sum_s d^{\pi}(s) \sum_a \pi(s,a)[Q^{\pi}(s,a) - ...