Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

226 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
19
votes
0answers
441 views

Formal proof of vanilla policy gradient convergence

So I stumbled upon this question, where the author asks for a proof of vanilla policy gradient procedures. The answer provided points to some literature, but the formal proof is nowhere to be included....
5
votes
0answers
141 views

How to predict advantage value in deep reinforcement learning

I'm currently working on a collection of reinforcement algorithms: https://github.com/lhk/rl_gym For deep q-learning, you need to calculate the q-values that should be predicted by your network. There ...
4
votes
0answers
272 views

Optimal implementation of vanilla DQN loss in Keras

I've implemented vanilla DQN for continuous/non-images (no CNN) states in keras. But, I'm not sure if my implementation of the loss computation is optimal. For reminder the loss is defined as : $loss=...
3
votes
0answers
28 views

Reinforcement Learning applied to Optimisation Problem

Problem Statement: We are given an optimisation problem; with production centres, source airport, destination airports, transfer points and finally delivered to the customers. This is better explained ...
3
votes
0answers
41 views

NN training with repetitive features

I posted the question also on ai.stackexchange but it didn't get any answers so I though I could try here. Here is a copy paste: Let's say you are training a NN in a RL setting where the state (i.e. ...
3
votes
1answer
43 views

What's the definition of retraining?

In transfer learning, we always use new data to retrain the pre-trained model. But, what is the specific and official definition of retraining? Or what papers mentioned this definition, in transfer ...
3
votes
0answers
142 views

Learn large, variable-size action space for Diplomacy game

I am making an environment using OpenAI gym for Diplomacy, and making an AI for it. In Diplomacy, a player has many units, and each unit has a number of moves available to it. Therefore, the player'...
3
votes
0answers
159 views

Reinforcement Learning using PPO2 in openai gym retro, mario not learning the clear the easy episode

I am training mario game in retro using ppo2 baselines for some time. I have tried level3 and level1 too. But even after full training when I play using saved checkpoints, the mario is not able to ...
3
votes
1answer
848 views

DQN fails to find optimal policy

Based on DeepMind publication, I've recreated the environment and I am trying to make the DQN find and converge to an optimal policy. The task of an agent is to learn how to sustainably collect apples ...
3
votes
0answers
671 views

Deep Reinforcement Learning for dynamic pricing

I am trying to implement a Deep Q Network model for Dynamic pricing in Logistics. I can define State Space (Origin, Destination, type of the shipment, customer, Type of the product, Commodity of the ...
3
votes
2answers
306 views

Reward function to avoid illegal actions, minimize legal action and learn to win - Reinforcement Learning

I'm currently implementing PPO for a game with the following characteristics: Observation space: 9x9x(>150) Action space: 144 In a given state, only a handful of actions (~1-10) are legal The state ...
3
votes
2answers
161 views

Is there a rule of thumb when designing neural network in deep reinforcement learning?

In deep learning, we can assess model's performance with loss function value and improve model's performance with K-fold cross-validation and so on. But how can we design and tune neural network used ...
3
votes
1answer
336 views

Time horizon T in policy gradients (actor-critic)

I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture. At the bottom of that slide, the gradient of the expected sum of rewards ...
3
votes
0answers
859 views

Defining State Representation in Deep Q-Learning

So I am having difficulty difficulty figuring out exactly how I want to represent my environment state in my Deep Q-learning problem. Premise: There is a 2D grid space of which an agent needs to ...
3
votes
1answer
229 views

Choosing a right algorithm for template-based text generation

I am doing a text generation project -- the task is to basically represent the statistical data in a readable way. The way I decided to go about this is template-based: each data type has a template ...
3
votes
0answers
142 views

Graphical results of Q-Learning: is improvement possible by parameter tweaking?

From left to right: Maximum Q value for action selection (averaged) Train error (averaged) Reward from environment (averaged) I run double Q-learning. A behavioral policy is ε-greedy, ε constant ...
3
votes
0answers
358 views

What is significance of Colour-digit MNIST game in paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning?

My question is regarding the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning. Can anyone explain what is the significance of Colour-digit MNIST game in the paper? I ...
2
votes
0answers
12 views

Building a simulator for continuous state, discrete action reinforcement learning

I am trying to build a simulator that optimizes the performance and temperature of a device. I want the device to perform well, but without making the device too hot. If the device becomes too hot, I ...
2
votes
1answer
48 views

Slow keras fit method with 100x100 array, how can I make it faster?

how can I make this training faster ? when I call the fit method on a 100 x 100 matrix goes very slow my model it's a sequential ...
2
votes
0answers
84 views

Offline/Batch Reinforcement Learning: Doubly Robust Off-policy Estimator takes huge values

Context: My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, etc.). It is too costly for us to emulate agents. We ...
2
votes
0answers
31 views

Entropy-regularized RL (G-learning) vs. IRL (Inverse Reinforcement Learning)

What are the differences between entropy-regularized RL (G-learning) and IRL (Inverse Reinforcement Learning)? and how are they applied to actual problems (besides stand-alone Markov decision ...
2
votes
1answer
20 views

Representing a 2d-grid around an agent

I'm trying to train a neural network-based model to play a game similar to Pac-Man, except there's no maze. i.e., the player is in a 2-dimensional grid, with dots of food in some locations, and the ...
2
votes
1answer
55 views

Policy Gradient not "learning"

I'm attempting to implement the policy gradient taken from the "Hands-On Machine Learning" book by Geron, which can be found here. The notebook uses Tensorflow and I'm attempting to do it with PyTorch....
2
votes
0answers
37 views

Find highest reward for epsilon-greedy bandit program

I started to learn reinforcement learning, the first example is handling bandit program using epsilon-greedy method, In this example, there are three bandit machines used, the output is the mean value ...
2
votes
0answers
42 views

Actions taken by agentn/ agent performance not improving

Hi I am trying to develop an rl agent using PPO algorithm. My agent takes an action(CFM) to maintain a state variable called RAT in between 24 to 24.5. I am using PPO algorithm of stable-baselines ...
2
votes
0answers
53 views

Incentivizing curiosity in a sparse reward environment

I'm quite new to reinforcement learning, but have been exploring different kinds of architectures (DQN, dueling DQN, actor critic, etc.) and evaluating their ability to solve certain problems. The ...
2
votes
1answer
106 views

Maximum Entropy Policy Gradient Derivation

I am reading through the paper on Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review by Sergey Levine. I am having a difficulty in understanding this part of the ...
2
votes
0answers
15 views

Autonomous learning - chatbots

My chatbots need to be trained when we get new data or feedbacks from users. Can someone provides ways how these chatbots can learn on themselves and become intelligent day by day? Some of techniques ...
2
votes
1answer
245 views

Guidelines to debug REINFORCE-type algorithms?

I implemented a self-critical policy gradient (as described here), for text summarization. However, after training, the results are not as high as expected (actually lower than without RL...). I'm ...
2
votes
1answer
20 views

Is it beneficial to use the last $N$ data points to train an RL agent?

Given that an environment in reinforcement learning is a Markov Decision Process (MDP), are there ever any cases where it is beneficial (or indeed where it makes sense) to use the last $N>1$ data ...
2
votes
1answer
68 views

What is the best way Reinforcement learning, RNN or others to predict the best action we have to take to maximize sales?

I have a dataset composed of few features : customerId, actionDay1, SalesDay1, actionDay20, SalesDay20, actionDay30, SalesDay30 action can be : call email face ...
2
votes
0answers
24 views

Which one of these is the most efficient way to model training data for a neural network that will play a snake-like game?

I am building an AI using a neural network that will play Tron against a human player. The game consists of a board with fixed width and height where each player can move at any direction (except for ...
2
votes
0answers
267 views

Policy Gradient methods not converging to useful mean values

I am getting familiar with Policy Gradient methods, specifically Advantage Actor Critic (A2C). My target problem use clipped continuous state and action spaces and I have therefore been training my ...
2
votes
1answer
272 views

Policy gradient/REINFORCE algorithm with RNN: why does this converge with SGM but not Adam?

I am working on training RNN model on caption generation with REINFORCE algorithm. I adopt self-critic strategy (see paper Self-critical Sequence Training for Image Captioning) to reduce the variance. ...
2
votes
1answer
144 views

Reinforcement Learning on real time data over a web server

Question: is it possible to implement a reinforcement learning model over a NodeJS server? This server would be receiving binary forms of data (open /close; yes/no) in real time. The objective for ...
2
votes
1answer
313 views

Problem when cherry picking actions - Proximal Policy Optimization

I am using the implementation of PPO2 in stable-baselines (a fork of OpenAI's baselines) for a Reinforcement Learning problem. My observation space is $9x9x191$ and my action space is $144$. Given a ...
2
votes
0answers
708 views

Pytorch: How to create an update rule the doesn't come from derivatives?

I want to implement the following algorithm, taken from this book, section 13.6: Here, the neural networks' outputs are $V(S, w)$ and $\pi(A|S,\theta)$, parameterized by $w$ and $\theta$ respectively....
2
votes
1answer
159 views

How does Q-Learning deal with mixed strategies?

I'm trying to understand how Q-learning deals with games where the optimal policy is a mixed strategy. The Bellman equation says that you should choose $max_a(Q(s,a))$ but this implies a single unique ...
2
votes
1answer
166 views

Policy gradient - and auto-differentiation (Pytorch/Tensorflow)

In policy gradient, we have something like this: Is my understanding correct that if I apply log cross-entropy on the last layer, the gradient will be automatically calculated as per formula above?
2
votes
1answer
680 views

Policy gradient: why does this converge with Adam and not SGD?

I am looking into policy gradient methods. I stumbled into this implementation: https://gist.github.com/calclavia/cfcd41ad4e47d7b9b6ab8af15410747a It uses a Nesterov Adam optimizer. If I run it, it ...
2
votes
1answer
77 views

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )? I want to replace the DNN section of Qlearning with a RF or SVR but the problem is that there is no clear ...
2
votes
0answers
223 views

Updating weights python for REINFORCE policy gradient method

All, I am trying to implement REINFORCE(williams) algorithm. This is a policy gradient reinforcement learning algorithm. I am using python, and hope to use keras. The pseduocode I am using is as ...
2
votes
0answers
181 views

Epoch greedy algorithm for contextual bandits

I'm reading the following paper on the epoch greedy algorithm for the contextual bandits problem. I have two questions http://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf I'm unsure ...
2
votes
0answers
1k views

Initial Q-values in Q-Learning

I am running a Q-learning algorithm with a finite time horizon. Are 'optimistic initial conditions' still preferred if there is a possibility that some states will not be visited multiple times? ...
1
vote
0answers
18 views

Jacks car rental problem: why deterministic policies?

In Sutton & Barto Book: Reinforcement Learning: An Introduction, there is the following problem: I have this question: why are the policies to be considered here are deterministic?
1
vote
0answers
16 views

Which ML to use for search suggestion?

Problem: I want to create a program to organize text information and fast access to relevant documents. I would like to train a ML model to analyse the current situation and to suggest the next ...
1
vote
0answers
14 views

Reinforcement learning in real world without OpenAI Gym

I have been searching for reinforcement learning libraries or examples that aren't using a simulated environment like OpenAI Gym. I havn't been successful at all until yet and I would really ...
1
vote
0answers
12 views

How to understand expected value function in a stochastic policy setting?

I am currently reading "Deep Reinforcement Learning with Python" by Sudharsan Ravichandiran. I am still on the first chapter of Introduction and understanding the most basic concepts. He has ...
1
vote
1answer
20 views

Difference between regret and pseudo-regret definitions in multi-armed bandits

I posted this question Cross Validated, but didn't get any answer. So I am posting it here too, as the question is very relevant to machine learning I am following the book Bandit Algorithms. In page ...
1
vote
0answers
74 views

Soft actor-critic reinforcement learning for 100x100 maze environment

I am doing a project which requires a soft actor-critic reinforcement learning agent to learn how to reach a goal in a 100x100 maze environment as the one below: The state space is discrete and only ...

1
2 3 4 5