# Help with transitioning an existing DQN into a DRQN

Hi Data Science Stack Exchange community,

To preface this post, please let me know if I need to clarify any details to receive help and/or guidance. I am new to posting on Data Science Stack Exchange and still consider myself a novice in the deep RL domain.

What I need help with is transitioning an existing DQN into a DRQN. The DQN architecture and the environment that it learns comes directly from this paper https://arxiv.org/pdf/1810.04244.pdf

To briefly summarize the paper, the author proposes a DQN network as a controller to guide fixed winged aircrafts to follow the evolution of a spreading wildfire (grid environment). The same DQN can be used for the both aircrafts. The inputs are follows:

To briefly summarize the paper, the author proposes a DQN network as controller to guide fixed winged aircrafts to follow the evolution of a spreading wildfire (grid environment). The same DQN can be used for the both aircrafts. The inputs are follows:

A vector of size 5:

• bank angle of ownship
• distance to other aircraft
• bearing angle to other aircraft relative to current heading direction of the ownship
• heading direction of the ownship
• bank angle of other aircraft

A tensor of dimensions 2x100x100. Two 100x100 dimension matrix stack upon each other.

• First matrix is a binary belief map of the fire. Indicating whether or not the cell is believed to be on fire.
• Second Matrix is a time elapsed or last visited map. Cell values in this matrix range from 0 to 250. A value 0 indicates the cell was just visited in most recent timestep. A value of 250 indicates that a cell has never been visited during the rollout or hasn’t been visited in over 250 time steps. If a cell is not visited, then the belief that the cell is on fire remains the same, and the time since last visited is incremented by 1 until reaching a maximum of 250 time steps. As the aircraft flies around the grid, each cell within a 10 cell radius of the aircraft is said to be visited.

The action space is a simple binary value. 0 to adjust the UAVs bank angle by -5 degrees and 1 to adjust the UAVs bank angle by +5 degree.

I was able to successfully replicate the DQN proposed in the paper and environment. See GIF: https://github.com/ajcantor1/wildfire_uav_surveillance_rl/blob/main/dqn_example.gif

The DRQN fails to converge to a meaningful solution or anything that resembles the behavior of the DQN.

With the DQN proposed in paper working correctly, it is probably safe to assume that the issue exist within the details of my DRQN implementation.

Below are major details of transitioning my DQN code to DRQN (possible culprits):

1. Network Architecture doesn’t deviant too much from the original DQN. I added an LSTM at the second to last layer (before the final dense output).

1. In the vanilla DNQ, replay memory implementation is simple. There is delineation whether an experience is part of a sequence/episode. During training, I select experiences randomly regardless of the episode they belong to. The sample’s dimension for a round training is the BATCH SIZE multiplied by the respective dimensions of the states, actions, rewards, and next state. A DRNQ requires a tad more overhead with maintaining episodes. For training the DRNQ, a sample consist of randomly selected episodes. Then for each episode, I randomly selected a contiguous block of a fixed SEQ_LENGTH. This adds an extra dimension to the sample, which leads to my third point.

2. The biggest caveat in my opinion with adding the LSTM layer is the dimensions. For the LSTM constructor, I set batch_first to True. For the training forward pass I condense the BATCH and SEQUENCE dimension into a single dimension. After the concatenation step, I reshape the data accordingly (BATCH_SIZE, SEQ_LENGTH, concatenation size). There is potential that I may have screwed this step up and incorrectly using pytorch view method?

  def forward(self, belief_map, state_vector, hidden = None, training=False):

if training:
state_vector = state_vector.view(-1, 5)
belief_map   = belief_map.view(-1, 2, 100, 100)

fc1_out = self.fc1(state_vector)
#print(f'fc1_out.shape: {fc1_out.shape}')
conv_out = self.conv(belief_map)
#print(f'conv_out.shape: {conv_out.shape}')
flatten_out = torch.flatten(conv_out, 1)
#print(f'flatten_out.shape: {flatten_out.shape}')
fc2_out = self.fc2(flatten_out)
#print(f'fc2_out.shape: {fc2_out.shape}')
concatenated = torch.cat((fc1_out, fc2_out), dim=1)

fc3_out = self.fc3(concatenated)

if training:
fc3_out = fc3_out.view(BATCH_SIZE, SEQ_LENGTH, 200)

if hidden is None:
lstm_out, hidden_out = self.lstm(fc3_out)
else:
lstm_out, hidden_out = self.lstm(fc3_out, hidden)

return self.fc4(lstm_out), hidden_out

1. The evaluation step/computing does not deviant too much from the DQN. The only difference is “unrolling” the BATCH and SEQUENCE dimension to obtain all predicted q values.
belief_maps, state_vectors, actions, next_belief_maps, next_state_vectors, rewards =
memory.sample(BATCH_SIZE, SEQ_LENGTH)

policy_output, _ = policy_net(belief_maps.cuda(), state_vectors.cuda(), training=True)

state_action_values = policy_output.view(BATCH_SIZE*SEQ_LENGTH, -1).gather(1, actions.view(BATCH_SIZE*SEQ_LENGTH, -1).cuda())

state_action_values = state_action_values.view(BATCH_SIZE, SEQ_LENGTH)

target_out, _ = target_net(next_belief_maps.cuda(), next_state_vectors.cuda(), training=True)
next_state_values = target_out.max(2)[0].view(BATCH_SIZE, SEQ_LENGTH).detach()
expected_state_action_values = (next_state_values * GAMMA) + rewards.cuda()
criterion = nn.SmoothL1Loss().to(device)
loss = criterion(state_action_values, expected_state_action_values)


For those who are interested and somehow generous enough to review my entire notebook: https://github.com/ajcantor1/wildfire_uav_surveillance_rl/blob/main/train_drqn.ipynb