2
$\begingroup$

I'm new to reinforcement learning so please bear with me. I'm training an agent to play ms-Pacman using the actor-critic method.

Below are the results of a couple of runs, in both graphs the orange line in the average of the previous 100 values. The left graph is the episode duration and the right is the reward earnt in each episode.

In both the runs, the agent steadily learns and then collapses, and is unable to recover. Has the agent reached it's maximum potential with this method or is this a case of overfitting?

Hyperparameters for both the runs:

  • Batch Size: 128
  • Replay Memory Buffer: 500,000
  • Epsilon Minimum: 0.1
  • Epsilon Maximum: 1.0
  • Epsilon minimum is reached at around 2,000,000 steps (approx episode 3,000)

enter image description here

enter image description here

Improve this question
$\endgroup$
9
  • $\begingroup$ Could you clarify if you are using Actor Critic or Deep Q? The hyperparameters you are mentioning are for Q learner and not an AC method. Which algorithm are you using precisely? Not quite sure what the orange line is. Is it like smoothed duration? $\endgroup$
    – Constantinos
    Jun 26, 2020 at 5:12
  • $\begingroup$ I may be confusing two things here. I do have a Q-learner. But I also have two policies with one called the actor and the other the critic. The actor plays the game and is trained from a sample in the memory buffer. Now and again the weights from the actor are transferred to the critic. The orange line is essentially a moving average. Hopefully, this clears a few things up? $\endgroup$
    – Harpal
    Jun 26, 2020 at 10:48
  • $\begingroup$ 1/2 First a kind suggestion, as I suggest to most people starting with RL: Start from the beginning :) Learn what is an Actor, what is a Critic and what is a policy and what you are trying to optimize. Find implementations that work and achieve good score (check what is the max score that the domain is considered solved and how many iterations it takes). Actor Critic outputs ONE policy. It is your implementation that has 2 networks. Get a grasp of what the Deep Q learning algorithm is actually doing. Then move on with AC methods. $\endgroup$
    – Constantinos
    Jun 26, 2020 at 21:20
  • $\begingroup$ 2/2 Eventually, check some good, simple and optimized for readability purposes implementations of DeepQ and A2C/A3C algorithms. There is no need for you to train 2 separate networks (Actor and Critic). You can use one with different heads at the last fully connected layer of the network. Hope these help :) $\endgroup$
    – Constantinos
    Jun 26, 2020 at 21:24
  • $\begingroup$ Thanks for the suggestions. I'm probably getting ahead of myself here. I was following the example in the hands-on machine learning book. The notebook for it can be found here: github.com/ageron/handson-ml/blob/master/… (go to heading "Learning to Play MsPacman Using the DQN Algorithm"). The author does state there is an error in that it should not be called an actor and critic but rather online and targen DQNs $\endgroup$
    – Harpal
    Jun 27, 2020 at 13:44

1 Answer 1

Reset to default
1
$\begingroup$

There are lots of things that might lead to this performance. With the very limited information provided I can only guess and suggest the reasons.

I have personally used lots of Pacman variations (including this one) with RL models successfully. My first initial guess is that you are stopping exploration too early. Pacman environments are very hard to solve (up to the last level). You can get a reasonable performance though within a couple of days training (depending on your system). I highly doubt that the agent underfits/overfits as it will take time for the agent to reach a good score. Also underfitting/overfitting would be better observable when you test your agent (you do not update weights). I would let exploration to become minimum once the agent reaches the last episode. Then I would test the current state of the agent with very little or none amount of exploration (full exploitation).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.