I'm not an expert in machine learning. I'm trying to make an agent learn to play in a Gym environment that I created. I'm not asking for help in debugging the code. I'm asking for help in interpreting the results of my simulations.
I'm tuning a DQN agent for a gym environment that I made myself. It is an environment with two players where one is a DQN agent training during the simulation and the other is a random agent that sample a random action from the action space.
I've used a Q network composed only by dense layers with ReLU activation in the hidden layers and linear activation in the output layer. The number of units in the hidden layers are 512, 512, 256, 128.
The environment hasn't a fixed duration: the episode ends when one of the two agents wins or when the time ends. For this reason, a fixed reward of -1 is provided at each time step and a sparse reward of plus or minus 200 is provided if one of the agents wins or loses the episode before the time ends.
The important parameters of the DQN agent (ask me for more if needed) are:
- policy used when choosing the action to take: Boltzmann distribution
- optimizer: Adam with
The results are based on 5 simulations of 500000 timesteps each. Each plot shows the mean over the 5 simulations in white and the standard deviation in blue. Finally, I've plotted the moving average over 10 episodes to make them smooth enough.
Of course, the agent learns nothing at all. This isn't what bothers me at the moment. I desire help on how to interpret the other graphs. In all other environments that I used this DQN agent the graphs of mean Q, loss, and mean absolute error, are much more different. In particular, loss and mean absolute error tended to increase until the reward had grown enough and only then they started to slowly decrease. On the other hand, the Q value usually decreased in the first episodes and then stabilize in the last episodes (in some cases showing a slight increase).
What I'm thinking is that the Q network stabilizes too soon (before actually learning anything), thus the first hyperparameter to tune should be the learning rate of the optimizer. Is this interpretation correct? Should I've noticed something else?