I am doing a project where I have multiple soft actor-critic sub-agents learning at the same time in an environment using shared experiences. Each sub-agent selects an action using their own policy, the overall agent combines these into a single action choice which gets submitted as the action. The environment gives each agent an individual reward for this action. Each agent then adds this (state, action, next_state, reward, done) tuple to its replay buffer which is later used to update it's policy.

To show a minimal example, I do something like:

actions = [agent.select_action(state) for agent in agents]
selected_action = policy(actions)
next_state, rewards, done, info = env.step(selected_action)
for agent in agents: 
   agent_reward = rewards[agent]
   agent.add_experience(state, selected_action, next_state, agent_reward, done)

To update each agent, I use a randomised replay buffer to select data:

data = self.replay_buffer.sample_batch(self.batch_size)

loss_q, q_info = self.compute_loss_q(data)

loss_pi, pi_info = self.compute_loss_pi(data)

My question is, does using an experience which was never actually selected by the policy network of the agent corrupt the learning of that agent?

My intuition is that doing so might incorrectly change weights in the network since those weights never lead to the action that is ultimately seen in the replay buffer. If so, is there a way to effectively share experiences between different agents?

Any help is greatly appreciated :)


In short, the answer is yes, your intuition is correct. The problem you are trying to solve is a Multiagent Reinforcement Learning problem and single-agent approaches don't work well here. One of the main obstacles is the non-stationary transitions of the environment's states. There are various settings (e.g. competitive and collaborative) and training techniques (decentralized and centralized) or combinations of them. You also need to be very careful how you will design the state space and the reward function of your problem.

One approach that is commonly used in AC architectures is to use a centralized critic and decentralized actors. In other words, critic receives the full state of the environment whereas the actors receive the local state representation.

For your specific problem, I suggest you to take a look at this review of Cooperative MARL, especially p. 16. 17 which reviews DQN and the role of the experience replay and then discusses centralized critic.

  • $\begingroup$ Thank you! I think that makes sense. I will have a look at the paper. When you say full state for the critic, does that refer to the combined state of all the individual actors? $\endgroup$ Aug 20 '21 at 11:22
  • $\begingroup$ You are welcome! Not necessarily. In general if you have a gridworld the full state representation would be the position of each agent and obstacles. The implementation is up to you. I recommend you to take a look at this (quite robust and clean) implementation: arxiv.org/abs/2103.01955. Read also 3.2 to get an idea of a successful approach regarding the full state representation. $\endgroup$ Aug 20 '21 at 18:10
  • $\begingroup$ Fantastic! Thank you so much $\endgroup$ Aug 21 '21 at 1:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.