Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Filter by
Sorted by
Tagged with
2
votes
1answer
39 views

Does convergence equal learning in Deep Q-learning?

In my current research project I'm using the Deep Q-learning algorithm. The setup is as follows: I'm training the model (using Deep Q-learning) on a static dataset made up of experiences extracted ...
0
votes
1answer
12 views

Reinforcement Learning model always gives different output

I am trying to build a reinforcement learning model for hardware capacity optimisation. The state of the model would input like CPU capacity utilisation, memory utilisation. The model is supposed to ...
3
votes
1answer
68 views

Confusion about the Bellman Equation

In some resources, the belman equation is shown as below: $v_{\pi}(s) = \sum\limits_{a}\pi(a|s)\sum\limits_{s',r}p(s',r|s,a)\big[r+\gamma v_{\pi}(s')\big] $ The thing that I confused is that, the $\pi$...
0
votes
0answers
17 views

Steps Needed To Create A Large Personal Recommender System [closed]

I am currently going through the process of creating a personal recommender system (Contextual Bandits) for users for Movie Recommendations. The data I have available: Context: UserNumber (few ...
0
votes
0answers
29 views

How to define observation space for a custom environment in OpenAI gym?

I am a newbie in OpenAI gym. I need help to define the observation space for a sensor network environment. Suppose, there are N sensor nodes in a network. Each node has three features. So, I have ...
0
votes
1answer
15 views

Which solutions are there for RL agents when not all actions are always available?

I'm working in an RL environment where not all actions are always available. In this case, depending on the state where the environment is at, some of the actions are not available for the agent to ...
0
votes
0answers
10 views

How can my loss be stable while the gradient keeps growing?

I have been working on an Offline/Batch Reinforcement Learning problem where I am using a BCQ-DDQN model as a Q-table. The model input is a state of 8 dimensions, and the output is a vector of Q-...
1
vote
0answers
17 views

Is “nb_steps_warmup” set for each episode or globally?

When I configure a DQN agent, nb_steps_warmup can be set. Is this parameter set for each episode or once globally? What I am trying to ask is, imaging I have a game ...
0
votes
0answers
16 views

RL PPO Algorithm: Understanding the Value Function Loss term in PPO by OpenAI

In the Schulman 2017 PPO Paper, there is a value function loss term in the final loss in equation 9, where they state that the value function loss is the MSE of the target value and predicted value. ...
0
votes
1answer
36 views

How to report results of RL with high variance?

I run Q-learning and SARSA algortihms on the same problem but the results fluctuate heavily and when I draw them, there is no smooth graph. How should I repost the results? I run algorithms for 500 ...
1
vote
0answers
29 views

Effects of slipperiness in OpenAI FrozenLake Environment

I am trying to wrap my head around the effects of is_slippery in the open.ai FrozenLake-v0 environment. From my results when ...
1
vote
0answers
10 views

Reinforcement learning example when the action is a matrix

I am working on solving a problem with reinforcement learning which has to find the optimal matrix that maximize the reward. I am not able to see how I can formulate this problem as I have practiced ...
0
votes
0answers
12 views

How to apply policy gradient to discrete combinatorial action space?

I'm new to RL techniques, so this may be a dumb question. It is a combinatorial scenario. For example, the action space is 5*3*4, i.e., I have to make a choice in each of the 3 sub-action sets, which ...
0
votes
0answers
6 views

How to get accurate estimates on Neural Networks Hessian?

I need to get not only accurate estimates on the neural network output itself but also on its second order derivatives in order to use the NN for optimization problems. With Adam optimizer I can't get ...
2
votes
0answers
35 views

Offline/Batch Reinforcement Learning: Doubly Robust Off-policy Estimator takes huge values

Context: My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, etc.). It is too costly for us to emulate agents. We ...
0
votes
1answer
38 views

Why MADDPG rather than taking all cooperating agents as a single meta-agent?

Since MADDPG uses a centralized critic for training, why not simply treat all cooperating agents as a single meta-agent with a concatenated observation space and a concatenated action space? In my ...
0
votes
0answers
13 views

Should I save DQN parameters for performance spikes?

I am currently running a DQN on the Atari Breakout OpenAI Gym. Over time the total number of points per episode is increasing, but occasionally the performance spikes to achieve 200+ points on a ...
0
votes
0answers
20 views

What sort of models work for unsupervised reinforcement learning, or is deep learning the way?

I'm setting out on an adventure to automate the statuses of the lights around my home. The lights should have different brightness in the range [0, 100] depending on some factors, which I have boiled ...
9
votes
2answers
1k views

What is the difference between active learning and reinforcement learning?

From Wikipedia: Active learning is a special case of machine learning in which a learning algorithm can interactively query a user (or some other information source) to label new data points with the ...
1
vote
0answers
40 views

Difference between Q-learning and G-learning in Reinforcement Learning?

What is the difference between Q-learning and G-learning in Reinforcement Learning? Please explain with formulas. An example source: Instead of relying on a utility of consumption, we present G-...
2
votes
0answers
18 views

Entropy-regularized RL (G-learning) vs. IRL (Inverse Reinforcement Learning)

What are the differences between entropy-regularized RL (G-learning) and IRL (Inverse Reinforcement Learning)? and how are they applied to actual problems (besides stand-alone Markov decision ...
0
votes
0answers
19 views

Epochs and other hyperparameters in Deep Q-Networks

I was wondering about hyperparameters used in Deep Q-Networks. Considering the use of replay memory and target network, together with the epsilon-greedy policy, are the number of epochs different of 1 ...
0
votes
0answers
33 views

KL divergence for exponential family distribution

In reinforcement learning, normal distribution is commonly used for continuous actions. I'm checking Pytorch's implementation for KL divergence between two normal distributions. I know it's impossible ...
0
votes
0answers
14 views

IRL with noisy observations?

I have a question about inverse reinforcement learning. As far as my research went, I found papers that deal with noisy observations of the state, but that's what the agent sees and the data scientist ...
1
vote
0answers
63 views

Different probability distributions for each number of a MultiDiscrete action space

I have made a custom gym environment, and I have a question regarding the actions. I use a MultiDiscrete action space, that is, it provides a list of integer ...
0
votes
0answers
23 views

Learning to Rank vs Reinforcement Learning in Information Retrieval - which one is preferable and why?

I am trying to create an information retrieval system which can benefit from user feedback (either implicit, through e.g., click-through data) or explicit (e.g., binary feedback on irrelevant ...
1
vote
1answer
46 views

Definition of the Q* function in reinforcement learning

I'm making my way through Sutton's Introduction to Reinforcement Learning. He gives the definition of the $q_*$ function as follows $$ q_*(a) = \mathbf{E}[R_t | A_t = a] $$ where $A_t$ is the action ...
0
votes
0answers
36 views

Reinforcement Learning with varying state space

I would like to ask a general question on reinforcement learning. Let $S = ([x_0, y_0], [x_1, y_1])$ be the states, where $lower \leqslant x_i, y_i \leqslant upper$. Is it possible (after training), ...
1
vote
0answers
32 views

Confidence in the rewards for a RL task

For a RL task that I am trying to solve, for which I train once per day, I have the rewards stored for each of those days, so that I can see the progress on daily basis. In the beginning of the ...
0
votes
0answers
118 views

Problem with AI gym from Google Collaboratory

I am a neophyte trying to learn and nowadays I am studying Raschka's 3rd edition Python Machine Learning. I am currently using Google Collaboratory and I am having an error when executing the ...
2
votes
0answers
14 views

Representing a 2d-grid around an agent

I'm trying to train a neural network-based model to play a game similar to Pac-Man, except there's no maze. i.e., the player is in a 2-dimensional grid, with dots of food in some locations, and the ...
1
vote
1answer
23 views

Matrix notation in Sutton and Barto

On pg. 206 of Barto and Sutton's Reinforcement Learning, there is a curious statement about the result of a scalar product: As I interpret it, A is the expectation of a scalar product of two d-...
2
votes
1answer
90 views

Question about AlphaGo Zero's Neural network architecture?

The following text is quoted from the AlphaGo Zero Paper 2017 from Nature. My question is regarding the eight features. The input to the neural network is a 19 × 19 × 17 image stack comprising 17 ...
1
vote
0answers
31 views

Reinforcement Learning for classification of a CSV file

I would like to try a Reinforcement Learning approach for a multi-label or binary classification of a CSV file. I know that Supervised Learning is probably easier and I have already tried a couple of ...
0
votes
1answer
371 views

DQN with decaying epsilon

I'm new to reinforcement learning. I'm studying DQN with decaying epsilon. I came across such example: EPISODES = 91 GAMMA = 0.2 EPSILON_DECAY = 0.999 MIN_EPSILON = 0.01 MAX_EPSILON = 1 My questions ...
0
votes
1answer
29 views

Different Initial Q-Values in Q-Learning

When working with Q-Learning, what is the difference between having a Q_0(a) with all values zero, random or optimistic?
0
votes
0answers
24 views

GAE - Understanding the TD based advantage function

The advantage function in GAE is defined as (eq 1) where (eq 2), The question is, in Eq 2, why is a value function (estimator, that needs to be trained) used at all? Especially when the value ...
0
votes
1answer
21 views

When should I use normal Q learning over a DQN?

From this article here, it says that using a tabular Q function is less scalable than a deep Q network. I assume that this means that the Q table approach works for some environments, but once they ...
0
votes
1answer
23 views

Do we need the outer discount term when implementing REINFORCE algorithm

I am learning the REINFORCE algorithm, which seems to be a foundation for other algorithms. I saw the $\gamma^t$ term in Sutton's textbook. But later when I watch Silver's lecture on this, there's no ...
1
vote
1answer
82 views

Deep Reinforcement Learning - mean Q as an evaluation metric

I'm tuning a deep learning model for a learner of Space Invaders game (image below). The state is defined as relative eucledian distance between the player and the enemies + relative distance between ...
1
vote
0answers
18 views

Learning the distribution of a continuous variable using LSTM

I am trying to implement the following paper : https://arxiv.org/pdf/2006.10701.pdf. In order, to estimate the priors of the hidden states which have continuous values, the authors use a LSTM. I have ...
1
vote
1answer
36 views

Proof of the connection between V and Q in Reinforcement Learning

I've been studying some basics RL in the SpinningUp materials. Is there any mathematical proof that $V^\pi(s) = E_{a \sim \pi} [Q^\pi(s, a)|s_0 = s]$ ?
1
vote
1answer
33 views

How to combine two differently scaled, but equally important “running” signals into a reward function?

I asked this question on Artificial Intelligence, but got no answer, so I am moving it here. I have two signals that I want to use to model a reward for a reinforcement learning algorithm. The first ...
0
votes
0answers
26 views

My Deep Q-Learning Network does not learn for OpenAI gym's cartpole problem

I am implementing OpenAI gym's cartpole problem using Deep Q-Learning (DQN). I followed tutorials (video and otherwise) and learned all about it. I implemented a code for myself and I thought it ...
0
votes
0answers
40 views

Deep Reinforcement Learning with Space Invaders

I want to better understand Deep Reinforcement Learning so I developed the Space Invaders game from scratch with Pygame. I have a fixed number of enemies (10). Instead of defining the states as a ...
1
vote
0answers
35 views

Reinforcement learning for turn-based AI

For a side project I'm trying to build a (simplified) AI for Heroes Of Might and Magic, using (as a starting point) deep Q-learning. But I'm having trouble to understand how the "state space"...
1
vote
0answers
35 views

In a double Deep Q network what would happen if we switch the roles of both networks

We normally use the online network for action selection and the target network for evaluation , would there be a difference if we switched the roles? Because in the case Of Double Q learning, we ...
1
vote
1answer
51 views

How is this score function estimator derived?

In this paper they have this equation, where they use the score function estimator, to estimate the gradient of an expectation. How did they derive this?
2
votes
0answers
22 views

How can I build a simulation environment that assess different risk policies? [closed]

I work in fin-tech and would like to build some sort of simulation program to assess how different inputs will impact net revenue. For example, if we create new policies based on ML scores, how would ...
0
votes
1answer
69 views

Monte Carlo for non-episodic tasks

In Sutton's textbook (Chapter 5) it says "To ensure that well-defined returns are available, here we define Monte Carlo methods only for episodic tasks". Can someone explain what exactly ...

1
2 3 4 5
10