Skip to main content

Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Filter by
Sorted by
Tagged with
0 votes
1 answer
373 views

Actor Network Target Value in A2C Reinforcement Learning

In DQN, we use; $Target = r+\gamma v(s')$ equation to train (fit) our network. It is easy to understand since we use the $Target$ value as the dependent variable like we do in supervised learning. I....
datatech's user avatar
3 votes
1 answer
51 views

Which Policy Gradient Method was used by Google's Deep Mind to teach AI to walk

I just saw this video on Youtube. Which Policy Gradient method was used to train the AI to walk? Was it DDPG or D4PG or what?
learner's user avatar
  • 33
0 votes
1 answer
849 views

Difference between $Q(s,a)$ ,$V^*(s)$ and $V^\pi(s)$ in Markov Decision Process?

I am new to RL and I am trying to understand how to find solutions of an MDP. This is what I understand so far -> since the nature of our environment is stochastic, at a state 's' if we take an ...
JANVI SHARMA's user avatar
1 vote
1 answer
29 views

Why is DDPG a Policy Gradient Method? [closed]

Why is DDPG a Policy Gradient Method even though it's actor does not output probability?
learner's user avatar
  • 21
0 votes
0 answers
90 views

Q values loss per episode and mean absolute error

I am new to deep reinforcement learning! I am following this code for my adaptation problem (doing actions) https://github.com/jaromiru/AI-blog/blob/master/CartPole-DQN.py I am wondering how I can ...
imen kanzali's user avatar
3 votes
3 answers
721 views

Is reinforcement learning a subset of unsupervised learning?

According to this article: Reinforcement learning on the other hand, which is a subset of Unsupervised learning ... How true is this statement? Is there any scholarly discussion/writing on the ...
Walid Mujahid وليد مجاهد's user avatar
0 votes
1 answer
40 views

How did the author got this final result from this Gaussian Distribution formula? [closed]

How did they got the final result?
learner's user avatar
  • 43
0 votes
1 answer
32 views

How to build a comunication interface between a reinforcement learning and a videogame?

I want to try to build a reinforcement learning model in order to play old arcade videogame, I know about projects in order to build AI for playing videogames, but I don't know how could I build an ...
Tlaloc-ES's user avatar
  • 337
0 votes
1 answer
362 views

Which policy gradient method is used for continuous action spaces?

Which policy gradient method is used that deals with continuous action spaces?
learner's user avatar
  • 43
1 vote
0 answers
265 views

Deep Q-learning in non-episodic tasks

I want to use Deep Q-learning (specifically DDQL by Hasselt et al. 2015, but it is the same principle) in a non-episodic task (continuing). I know that it is possible to use Q-learning in continuing ...
Tomás Lara's user avatar
1 vote
0 answers
271 views

DQN DDQN and 3DQN differences?

I'm doing a course on reinforcement learning, and one of our tasks is to implement an agent on the Lunar lander continuous V2 environment from openAI gym. In order to solve the continuous problem, I ...
user113367's user avatar
0 votes
2 answers
2k views

Understanding action space in stable baselines

I was trying to write reinforcement learning agent using stable-baselines3 library. The agent(abservations) method should return action. I went through different ...
Rnj's user avatar
  • 225
1 vote
0 answers
11 views

Implementing DQN - Pylessons.com article - Queries [closed]

I have followed the below link to implement DQN Algorithm https://pylessons.com/CartPole-reinforcement-learning/ Can someone explain me: Why do we need to have else condition in act function during ...
vimala's user avatar
  • 21
1 vote
0 answers
195 views

Reinforcement Learning reward is not converging to zero

I am new to reinforcement learning. I am trying to build an RL algorithm which will predict cloud hardware capacity required for an org in terms of compute, storage, memory. The algorithm which i have ...
kumar's user avatar
  • 79
1 vote
1 answer
2k views

Reinforcement Learning - PPO: Why do so many implementations calculate the returns using the GAE? (Mathematical reason)

There are so many PPO implementations that use GAE and do the following: ...
Johannes's user avatar
0 votes
0 answers
29 views

Blocked in my project with reinforcement learning.. any help?

I would like advice on a problem I'm trying to solve. I'm stuck... Here's an example: I have continuous data from a person such as what was eaten for a week (calories, carbohydrates, protein and fat) ...
wawouille's user avatar
3 votes
2 answers
513 views

How exactly does DQN learn?

I created my custom environment in gym, which is a maze. I use a DQN model with ...
Marci's user avatar
  • 31
1 vote
1 answer
33 views

Quick question on basic Basic concept of experience replay

Due to my admitted newbie's understanding on the field, I'm about to ask a dummy question. While sampling batches, for example experience replay buffer which contains number of samples, after getting ...
Quang Hung's user avatar
2 votes
1 answer
431 views

Does convergence equal learning in Deep Q-learning?

In my current research project I'm using the Deep Q-learning algorithm. The setup is as follows: I'm training the model (using Deep Q-learning) on a static dataset made up of experiences extracted ...
Aeryan's user avatar
  • 81
0 votes
1 answer
198 views

Reinforcement Learning model always gives different output

I am trying to build a reinforcement learning model for hardware capacity optimisation. The state of the model would input like CPU capacity utilisation, memory utilisation. The model is supposed to ...
kumar's user avatar
  • 79
3 votes
1 answer
121 views

Confusion about the Bellman Equation

In some resources, the belman equation is shown as below: $v_{\pi}(s) = \sum\limits_{a}\pi(a|s)\sum\limits_{s',r}p(s',r|s,a)\big[r+\gamma v_{\pi}(s')\big] $ The thing that I confused is that, the $\pi$...
datatech's user avatar
0 votes
1 answer
127 views

Which solutions are there for RL agents when not all actions are always available?

I'm working in an RL environment where not all actions are always available. In this case, depending on the state where the environment is at, some of the actions are not available for the agent to ...
Carlos Natalino's user avatar
1 vote
0 answers
118 views

Is "nb_steps_warmup" set for each episode or globally?

When I configure a DQN agent, nb_steps_warmup can be set. Is this parameter set for each episode or once globally? What I am trying to ask is, imaging I have a game ...
StefanOverFlow's user avatar
0 votes
1 answer
318 views

How to report results of RL with high variance?

I run Q-learning and SARSA algortihms on the same problem but the results fluctuate heavily and when I draw them, there is no smooth graph. How should I repost the results? I run algorithms for 500 ...
Aaron's user avatar
  • 201
1 vote
1 answer
423 views

Effects of slipperiness in OpenAI FrozenLake Environment

I am trying to wrap my head around the effects of is_slippery in the open.ai FrozenLake-v0 environment. From my results when ...
yudhiesh's user avatar
  • 213
2 votes
0 answers
125 views

Offline/Batch Reinforcement Learning: Doubly Robust Off-policy Estimator takes huge values

Context: My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, etc.). It is too costly for us to emulate agents. We ...
MetaHG's user avatar
  • 21
0 votes
1 answer
93 views

Why MADDPG rather than taking all cooperating agents as a single meta-agent?

Since MADDPG uses a centralized critic for training, why not simply treat all cooperating agents as a single meta-agent with a concatenated observation space and a concatenated action space? In my ...
Forrest Wei's user avatar
15 votes
2 answers
13k views

What is the difference between active learning and reinforcement learning?

From Wikipedia: Active learning is a special case of machine learning in which a learning algorithm can interactively query a user (or some other information source) to label new data points with the ...
Moradnejad's user avatar
1 vote
1 answer
665 views

Difference between Q-learning and G-learning in Reinforcement Learning?

What is the difference between Q-learning and G-learning in Reinforcement Learning? Please explain with formulas. An example source: Instead of relying on a utility of consumption, we present G-...
develarist's user avatar
2 votes
0 answers
46 views

Entropy-regularized RL (G-learning) vs. IRL (Inverse Reinforcement Learning)

What are the differences between entropy-regularized RL (G-learning) and IRL (Inverse Reinforcement Learning)? and how are they applied to actual problems (besides stand-alone Markov decision ...
develarist's user avatar
1 vote
0 answers
32 views

Epochs and other hyperparameters in Deep Q-Networks

I was wondering about hyperparameters used in Deep Q-Networks. Considering the use of replay memory and target network, together with the epsilon-greedy policy, are the number of epochs different of 1 ...
HenDoNR's user avatar
  • 23
1 vote
0 answers
268 views

Different probability distributions for each number of a MultiDiscrete action space

I have made a custom gym environment, and I have a question regarding the actions. I use a MultiDiscrete action space, that is, it provides a list of integer ...
Michail's user avatar
  • 11
1 vote
1 answer
361 views

Definition of the Q* function in reinforcement learning

I'm making my way through Sutton's Introduction to Reinforcement Learning. He gives the definition of the $q_*$ function as follows $$ q_*(a) = \mathbf{E}[R_t | A_t = a] $$ where $A_t$ is the action ...
marlineer43's user avatar
1 vote
0 answers
62 views

Confidence in the rewards for a RL task

For a RL task that I am trying to solve, for which I train once per day, I have the rewards stored for each of those days, so that I can see the progress on daily basis. In the beginning of the ...
ilirosmanaj's user avatar
2 votes
1 answer
43 views

Representing a 2d-grid around an agent

I'm trying to train a neural network-based model to play a game similar to Pac-Man, except there's no maze. i.e., the player is in a 2-dimensional grid, with dots of food in some locations, and the ...
Ram Rachum's user avatar
1 vote
1 answer
51 views

Matrix notation in Sutton and Barto

On pg. 206 of Barto and Sutton's Reinforcement Learning, there is a curious statement about the result of a scalar product: As I interpret it, A is the expectation of a scalar product of two d-...
corazza's user avatar
  • 113
2 votes
1 answer
182 views

Question about AlphaGo Zero's Neural network architecture?

The following text is quoted from the AlphaGo Zero Paper 2017 from Nature. My question is regarding the eight features. The input to the neural network is a 19 × 19 × 17 image stack comprising 17 ...
Qian Chen's user avatar
  • 121
2 votes
1 answer
5k views

DQN with decaying epsilon

I'm new to reinforcement learning. I'm studying DQN with decaying epsilon. I came across such example: EPISODES = 91 GAMMA = 0.2 EPSILON_DECAY = 0.999 MIN_EPSILON = 0.01 MAX_EPSILON = 1 My questions ...
Martin's user avatar
  • 123
1 vote
1 answer
458 views

Different Initial Q-Values in Q-Learning

When working with Q-Learning, what is the difference between having a Q_0(a) with all values zero, random or optimistic?
Giulia's user avatar
  • 11
0 votes
1 answer
295 views

When should I use normal Q learning over a DQN?

From this article here, it says that using a tabular Q function is less scalable than a deep Q network. I assume that this means that the Q table approach works for some environments, but once they ...
jasooney23's user avatar
0 votes
1 answer
72 views

Do we need the outer discount term when implementing REINFORCE algorithm

I am learning the REINFORCE algorithm, which seems to be a foundation for other algorithms. I saw the $\gamma^t$ term in Sutton's textbook. But later when I watch Silver's lecture on this, there's no ...
Star's user avatar
  • 3
1 vote
1 answer
354 views

Deep Reinforcement Learning - mean Q as an evaluation metric

I'm tuning a deep learning model for a learner of Space Invaders game (image below). The state is defined as relative eucledian distance between the player and the enemies + relative distance between ...
Yassine's user avatar
  • 35
1 vote
0 answers
28 views

Learning the distribution of a continuous variable using LSTM

I am trying to implement the following paper : https://arxiv.org/pdf/2006.10701.pdf. In order, to estimate the priors of the hidden states which have continuous values, the authors use a LSTM. I have ...
Oussa's user avatar
  • 11
1 vote
1 answer
117 views

Proof of the connection between V and Q in Reinforcement Learning

I've been studying some basics RL in the SpinningUp materials. Is there any mathematical proof that $V^\pi(s) = E_{a \sim \pi} [Q^\pi(s, a)|s_0 = s]$ ?
fig0's user avatar
  • 11
1 vote
1 answer
145 views

How to combine two differently scaled, but equally important "running" signals into a reward function?

I asked this question on Artificial Intelligence, but got no answer, so I am moving it here. I have two signals that I want to use to model a reward for a reinforcement learning algorithm. The first ...
tmaric's user avatar
  • 160
0 votes
0 answers
128 views

Deep Reinforcement Learning with Space Invaders

I want to better understand Deep Reinforcement Learning so I developed the Space Invaders game from scratch with Pygame. I have a fixed number of enemies (10). Instead of defining the states as a ...
Yassine's user avatar
  • 35
1 vote
0 answers
84 views

Reinforcement learning for turn-based AI

For a side project I'm trying to build a (simplified) AI for Heroes Of Might and Magic, using (as a starting point) deep Q-learning. But I'm having trouble to understand how the "state space"...
lezebulon's user avatar
  • 179
1 vote
0 answers
49 views

In a double Deep Q network what would happen if we switch the roles of both networks

We normally use the online network for action selection and the target network for evaluation , would there be a difference if we switched the roles? Because in the case Of Double Q learning, we ...
Chuki Bom's user avatar
1 vote
1 answer
100 views

How is this score function estimator derived?

In this paper they have this equation, where they use the score function estimator, to estimate the gradient of an expectation. How did they derive this?
adam's user avatar
  • 13
1 vote
0 answers
28 views

How can I build a simulation environment that assess different risk policies? [closed]

I work in fin-tech and would like to build some sort of simulation program to assess how different inputs will impact net revenue. For example, if we create new policies based on ML scores, how would ...
Kevin's user avatar
  • 11

1 2 3
4
5
13