Does a larger action space take longer to train an RL agent?

Question

I am playing around with the openai gym to try and better understand reinforcement learning. One agent parameter you can modify is the action space i.e. the specific actions an agent can take in an environment at each state e.g. "Left", "Right", "Up" or "Down" if the environment is a game with 4 discrete actions.

In my research, I have not found anywhere that explicitly states that an RL model, specifically PPO2, will take longer to train if the action space is larger.

All else being held the same; same data, same environment, same hyperparameters, same hardware, will a model with a larger action space (more possible actions) take longer to train (1 episode) than a model with a smaller action space?

(e.g. will an agent with 100 possible actions take longer to train 1 episode than an agent with 2 possible actions?)

Intuitively, I would have thought that the more actions an agent has, the more "choice" it has at each state, and therefore it would take longer in choosing in one of those actions. But again, I haven't found anything proving or disproving this.

Answer 1

I think there are two things in your question:

• The number of parameters of your network. So if you have more actions to predict the action layer of your network will have more parameters and it will take a little bit longer theoretically to update it (the time difference thought might be negligible depending your computational resources).
• The total training time. Yes, having more choices it will take longer for the algorithm to generate some good estimations of expected reward per action in order to lead to a satisfactory performance. If for Critic you are using a state value $V(s)$ instead of $Q(s,a)$ still your algorithm will take longer to explore the action space and map your input state to a probability distribution over your action space.