DQN not learning anything - Reinforcement Learning

Question

I am trying to train a DQN to play the 8puzzle game.

I have implemented a batched gameboards, so I am not using ReplayMemory.

Here's training process:

finished_counts = torch.tensor(0, device=device)
pbar = tqdm(range(int(1e4)))
for e in pbar:
    q_values = net(board) # board: Tensor with shape [10000, 3, 3]
    
    # choose action randomly or by q_values
    if np.random.rand() < 0.5:
        actions = torch.randint(0, 4, (board.size(0),)).to(device)
    else:
        actions = torch.argmax(q_values, dim=1)

    # reward: -100 for hit the wall, 10 for win, 0 for else
    reward, next_board = move(board, actions)

    next_q = net(next_board).detach()
    next_q[reward == 10] = 0 # ignore next_q if already win

    desired_q = q_values.clone().detach()
    distance = manhattan(next_board)
    desired_q[:, actions] = (reward - distance) + next_q[:, actions] * q_decay # q_decay = 0.9

    # set metrics
    pbar.set_postfix({
        'avg distance': f'{torch.mean(distance):.2f}',
        'finished': str(finished_counts.cpu().item()),
    })

    # compute loss and backwards
    # HuberLoss for criterion
    optimizer.zero_grad()
    loss = criterion(q_values, desired_q)
    loss.backward()
    optimizer.step() # Adam(lr=1e-4)

    board = next_board

    finished = distance == 0
    board = board[~finished]
    finished_counts += finished.sum()

What I obeserved is, the avg distance is remaing around 12.67, and finished_counts increasing in linear rate, which may attributed to random action.

My question is:

1. Is it ok to not use ReplayMemory?
2. Am I calculate the q_values and desired_q correct?
3. Why is the network not learning anything?

Answer 1

I figured it out a little with many experiments, here's some key rules needs to obey during building the model and training:

1. Keep positive samples and nagetive samples balanced.
   You have to let the Agent win, then it can learn a path leads to win. You can set to low difficulty initally, or guide it(by setting milestones of the game, and if it reaches the goal, give it a big reward). Sometimes you have to decrease the batch size to include more positive samples within a batch.
2. Do not torch.log(MSE(y_hat, y)), I don't know why.
3. Limit the Agent's max step, the random action will distrub Agent, pushing the game difficulty to higher, you don't want it to play a high difficulty game initially.
4. My code is wrong:

   next_q = net(next_board).detach()
   next_q = next_q.max(1)[0] # notice: use next_q's max value
   next_q[reward == GAME_WIN_REWARD] = 0 # ignore next_q if already win
   
   q_values = q_values.gather(1, actions.unsqueeze(1)).squeeze(1)
   distance = manhattan(next_board)
   desired_q = (reward - distance ** 2) + next_q * q_decay # I reinforce the effect of the distance
   

Here's how I train:
learning rate: 0.00005 (constant)
MAX_STEP: 10 * DIFFICULTY
BATCH_SIZE = 100

DIFFICULTY	EXPLORE RATE	EPISODE	mark
5	0.8	2e4	initally low difficulty
5	0.7	1e4
5	0.6	1e4
5	0.5	1e4
5	0.4	1e4
5	0.2	1e4
5	0	1e4	reinforce memory
6	0.3	1e4
6	0.1	1e4
6	0	1e4	reinforce memory
6	0.1	4e4	I found the Agent has bad performace in difficulty of 6
6	0	1e4
7	0.3	1e4
7	0.1	1e4
7	0	1e4