# Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

613 questions
Filter by
Sorted by
Tagged with
373 views

### Actor Network Target Value in A2C Reinforcement Learning

In DQN, we use; $Target = r+\gamma v(s')$ equation to train (fit) our network. It is easy to understand since we use the $Target$ value as the dependent variable like we do in supervised learning. I....
• 53
51 views

### Which Policy Gradient Method was used by Google's Deep Mind to teach AI to walk

I just saw this video on Youtube. Which Policy Gradient method was used to train the AI to walk? Was it DDPG or D4PG or what?
• 33
849 views

### Difference between $Q(s,a)$ ,$V^*(s)$ and $V^\pi(s)$ in Markov Decision Process?

I am new to RL and I am trying to understand how to find solutions of an MDP. This is what I understand so far -> since the nature of our environment is stochastic, at a state 's' if we take an ...
1 vote
29 views

### Why is DDPG a Policy Gradient Method? [closed]

Why is DDPG a Policy Gradient Method even though it's actor does not output probability?
• 21
90 views

### Q values loss per episode and mean absolute error

I am new to deep reinforcement learning! I am following this code for my adaptation problem (doing actions) https://github.com/jaromiru/AI-blog/blob/master/CartPole-DQN.py I am wondering how I can ...
721 views

### Is reinforcement learning a subset of unsupervised learning?

According to this article: Reinforcement learning on the other hand, which is a subset of Unsupervised learning ... How true is this statement? Is there any scholarly discussion/writing on the ...
40 views

### How did the author got this final result from this Gaussian Distribution formula? [closed]

How did they got the final result?
• 43
32 views

### How to build a comunication interface between a reinforcement learning and a videogame?

I want to try to build a reinforcement learning model in order to play old arcade videogame, I know about projects in order to build AI for playing videogames, but I don't know how could I build an ...
• 337
362 views

### Which policy gradient method is used for continuous action spaces?

Which policy gradient method is used that deals with continuous action spaces?
• 43
1 vote
265 views

### Deep Q-learning in non-episodic tasks

I want to use Deep Q-learning (specifically DDQL by Hasselt et al. 2015, but it is the same principle) in a non-episodic task (continuing). I know that it is possible to use Q-learning in continuing ...
1 vote
271 views

### DQN DDQN and 3DQN differences?

I'm doing a course on reinforcement learning, and one of our tasks is to implement an agent on the Lunar lander continuous V2 environment from openAI gym. In order to solve the continuous problem, I ...
2k views

### Understanding action space in stable baselines

I was trying to write reinforcement learning agent using stable-baselines3 library. The agent(abservations) method should return action. I went through different ...
• 225
1 vote
11 views

### Implementing DQN - Pylessons.com article - Queries [closed]

I have followed the below link to implement DQN Algorithm https://pylessons.com/CartPole-reinforcement-learning/ Can someone explain me: Why do we need to have else condition in act function during ...
• 21
1 vote
195 views

### Reinforcement Learning reward is not converging to zero

I am new to reinforcement learning. I am trying to build an RL algorithm which will predict cloud hardware capacity required for an org in terms of compute, storage, memory. The algorithm which i have ...
• 79
1 vote
2k views

### Reinforcement Learning - PPO: Why do so many implementations calculate the returns using the GAE? (Mathematical reason)

There are so many PPO implementations that use GAE and do the following: ...
• 11
29 views

### Blocked in my project with reinforcement learning.. any help?

I would like advice on a problem I'm trying to solve. I'm stuck... Here's an example: I have continuous data from a person such as what was eaten for a week (calories, carbohydrates, protein and fat) ...
513 views

### How exactly does DQN learn?

I created my custom environment in gym, which is a maze. I use a DQN model with ...
• 31
1 vote
33 views

### Quick question on basic Basic concept of experience replay

Due to my admitted newbie's understanding on the field, I'm about to ask a dummy question. While sampling batches, for example experience replay buffer which contains number of samples, after getting ...
431 views

### Does convergence equal learning in Deep Q-learning?

In my current research project I'm using the Deep Q-learning algorithm. The setup is as follows: I'm training the model (using Deep Q-learning) on a static dataset made up of experiences extracted ...
• 81
198 views

### Reinforcement Learning model always gives different output

I am trying to build a reinforcement learning model for hardware capacity optimisation. The state of the model would input like CPU capacity utilisation, memory utilisation. The model is supposed to ...
• 79
121 views

### Confusion about the Bellman Equation

In some resources, the belman equation is shown as below: $v_{\pi}(s) = \sum\limits_{a}\pi(a|s)\sum\limits_{s',r}p(s',r|s,a)\big[r+\gamma v_{\pi}(s')\big]$ The thing that I confused is that, the $\pi$...
• 53
127 views

### Which solutions are there for RL agents when not all actions are always available?

I'm working in an RL environment where not all actions are always available. In this case, depending on the state where the environment is at, some of the actions are not available for the agent to ...
1 vote
118 views

### Is "nb_steps_warmup" set for each episode or globally?

When I configure a DQN agent, nb_steps_warmup can be set. Is this parameter set for each episode or once globally? What I am trying to ask is, imaging I have a game ...
318 views

### How to report results of RL with high variance?

I run Q-learning and SARSA algortihms on the same problem but the results fluctuate heavily and when I draw them, there is no smooth graph. How should I repost the results? I run algorithms for 500 ...
• 201
1 vote
423 views

### Effects of slipperiness in OpenAI FrozenLake Environment

I am trying to wrap my head around the effects of is_slippery in the open.ai FrozenLake-v0 environment. From my results when ...
• 213
125 views

### Offline/Batch Reinforcement Learning: Doubly Robust Off-policy Estimator takes huge values

Context: My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, etc.). It is too costly for us to emulate agents. We ...
• 21
93 views

### Why MADDPG rather than taking all cooperating agents as a single meta-agent?

Since MADDPG uses a centralized critic for training, why not simply treat all cooperating agents as a single meta-agent with a concatenated observation space and a concatenated action space? In my ...
13k views

### What is the difference between active learning and reinforcement learning?

From Wikipedia: Active learning is a special case of machine learning in which a learning algorithm can interactively query a user (or some other information source) to label new data points with the ...
• 265
1 vote
665 views

### Difference between Q-learning and G-learning in Reinforcement Learning?

What is the difference between Q-learning and G-learning in Reinforcement Learning? Please explain with formulas. An example source: Instead of relying on a utility of consumption, we present G-...
• 214
46 views

### Entropy-regularized RL (G-learning) vs. IRL (Inverse Reinforcement Learning)

What are the differences between entropy-regularized RL (G-learning) and IRL (Inverse Reinforcement Learning)? and how are they applied to actual problems (besides stand-alone Markov decision ...
• 214
1 vote
32 views

### Epochs and other hyperparameters in Deep Q-Networks

I was wondering about hyperparameters used in Deep Q-Networks. Considering the use of replay memory and target network, together with the epsilon-greedy policy, are the number of epochs different of 1 ...
• 23
1 vote
268 views

### Different probability distributions for each number of a MultiDiscrete action space

I have made a custom gym environment, and I have a question regarding the actions. I use a MultiDiscrete action space, that is, it provides a list of integer ...
• 11
1 vote
361 views

### Definition of the Q* function in reinforcement learning

I'm making my way through Sutton's Introduction to Reinforcement Learning. He gives the definition of the $q_*$ function as follows $$q_*(a) = \mathbf{E}[R_t | A_t = a]$$ where $A_t$ is the action ...
1 vote
62 views

### Confidence in the rewards for a RL task

For a RL task that I am trying to solve, for which I train once per day, I have the rewards stored for each of those days, so that I can see the progress on daily basis. In the beginning of the ...
43 views

### Representing a 2d-grid around an agent

I'm trying to train a neural network-based model to play a game similar to Pac-Man, except there's no maze. i.e., the player is in a 2-dimensional grid, with dots of food in some locations, and the ...
• 255
1 vote
51 views

### Matrix notation in Sutton and Barto

On pg. 206 of Barto and Sutton's Reinforcement Learning, there is a curious statement about the result of a scalar product: As I interpret it, A is the expectation of a scalar product of two d-...
• 113
182 views

### Question about AlphaGo Zero's Neural network architecture?

The following text is quoted from the AlphaGo Zero Paper 2017 from Nature. My question is regarding the eight features. The input to the neural network is a 19 × 19 × 17 image stack comprising 17 ...
• 121
5k views

### DQN with decaying epsilon

I'm new to reinforcement learning. I'm studying DQN with decaying epsilon. I came across such example: EPISODES = 91 GAMMA = 0.2 EPSILON_DECAY = 0.999 MIN_EPSILON = 0.01 MAX_EPSILON = 1 My questions ...
• 123
1 vote
458 views

### Different Initial Q-Values in Q-Learning

When working with Q-Learning, what is the difference between having a Q_0(a) with all values zero, random or optimistic?
• 11
295 views

### When should I use normal Q learning over a DQN?

From this article here, it says that using a tabular Q function is less scalable than a deep Q network. I assume that this means that the Q table approach works for some environments, but once they ...
72 views

### Do we need the outer discount term when implementing REINFORCE algorithm

I am learning the REINFORCE algorithm, which seems to be a foundation for other algorithms. I saw the $\gamma^t$ term in Sutton's textbook. But later when I watch Silver's lecture on this, there's no ...
• 3
1 vote
354 views

### Deep Reinforcement Learning - mean Q as an evaluation metric

I'm tuning a deep learning model for a learner of Space Invaders game (image below). The state is defined as relative eucledian distance between the player and the enemies + relative distance between ...
• 35
1 vote
28 views

### Learning the distribution of a continuous variable using LSTM

I am trying to implement the following paper : https://arxiv.org/pdf/2006.10701.pdf. In order, to estimate the priors of the hidden states which have continuous values, the authors use a LSTM. I have ...
• 11
1 vote
117 views

### Proof of the connection between V and Q in Reinforcement Learning

I've been studying some basics RL in the SpinningUp materials. Is there any mathematical proof that $V^\pi(s) = E_{a \sim \pi} [Q^\pi(s, a)|s_0 = s]$ ?
• 11
1 vote
145 views

### How to combine two differently scaled, but equally important "running" signals into a reward function?

I asked this question on Artificial Intelligence, but got no answer, so I am moving it here. I have two signals that I want to use to model a reward for a reinforcement learning algorithm. The first ...
• 160
128 views

### Deep Reinforcement Learning with Space Invaders

I want to better understand Deep Reinforcement Learning so I developed the Space Invaders game from scratch with Pygame. I have a fixed number of enemies (10). Instead of defining the states as a ...
• 35
1 vote
84 views

### Reinforcement learning for turn-based AI

For a side project I'm trying to build a (simplified) AI for Heroes Of Might and Magic, using (as a starting point) deep Q-learning. But I'm having trouble to understand how the "state space"...
• 179
1 vote
49 views

### In a double Deep Q network what would happen if we switch the roles of both networks

We normally use the online network for action selection and the target network for evaluation , would there be a difference if we switched the roles? Because in the case Of Double Q learning, we ...
1 vote
100 views

### How is this score function estimator derived?

In this paper they have this equation, where they use the score function estimator, to estimate the gradient of an expectation. How did they derive this?
• 13
1 vote