I am doing a project where I have multiple soft actor-critic sub-agents learning at the same time in an environment using shared experiences. Each sub-agent selects an action using their own policy, the overall agent combines these into a single action choice which gets submitted as the action. The environment gives each agent an individual reward for this action. Each agent then adds this (state, action, next_state, reward, done) tuple to its replay buffer which is later used to update it's policy.
To show a minimal example, I do something like:
actions = [agent.select_action(state) for agent in agents]
selected_action = policy(actions)
next_state, rewards, done, info = env.step(selected_action)
for agent in agents:
agent_reward = rewards[agent]
agent.add_experience(state, selected_action, next_state, agent_reward, done)
To update each agent, I use a randomised replay buffer to select data:
data = self.replay_buffer.sample_batch(self.batch_size)
self.q_optimiser.zero_grad()
loss_q, q_info = self.compute_loss_q(data)
loss_q.backward()
self.q_optimiser.step()
self.pi_optimiser.zero_grad()
loss_pi, pi_info = self.compute_loss_pi(data)
loss_pi.backward()
self.pi_optimiser.step()
My question is, does using an experience which was never actually selected by the policy network of the agent corrupt the learning of that agent?
My intuition is that doing so might incorrectly change weights in the network since those weights never lead to the action that is ultimately seen in the replay buffer. If so, is there a way to effectively share experiences between different agents?
Any help is greatly appreciated :)
