Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

175 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
0answers
136 views

Formal proof of vanilla policy gradient convergence

So I stumbled upon this question, where the author asks for a proof of vanilla policy gradient procedures. The answer provided points to some literature, but the formal proof is nowhere to be included....
7
votes
2answers
113 views

Representing similar states in reinforcement learning?

Let's say I'd like to design a Q learning algorithm that learns to play poker. The number of different possible States is very large, but a lot are very similar: for example, if the initial state ...
5
votes
1answer
234 views

how to choose between discounted reward and average reward

how to select between average reward and discounted reward? And when average reward is more effective in comparison with discounter reward and when vice versa is correct? -Is is possible to use both ...
4
votes
2answers
83 views

Is there a rule of thumb when designing neural network in deep reinforcement learning?

In deep learning, we can assess model's performance with loss function value and improve model's performance with K-fold cross-validation and so on. But how can we design and tune neural network used ...
3
votes
1answer
155 views

Time horizon T in policy gradients (actor-critic)

I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture. At the bottom of that slide, the gradient of the expected sum of rewards ...
3
votes
0answers
308 views

Rainbow vs A3C …too unfair?

In Deep Mind's Rainbow paper, how come A3C algorithm be so slow? twice slower than DDQN... Was it trained on a single actor? :D It's on page 1 of the paper Wasn't A3C supposed to be something a lot ...
3
votes
0answers
88 views

Deep advantage learning: how to predict the value

I'm currently working on a collection of reinforcement algorithms: https://github.com/lhk/rl_gym For deep q-learning, you need to calculate the q-values that should be predicted by your network. ...
3
votes
1answer
91 views

Choosing a right algorithm for template-based text generation

I am doing a text generation project -- the task is to basically represent the statistical data in a readable way. The way I decided to go about this is template-based: each data type has a template ...
3
votes
0answers
132 views

Graphical results of Q-Learning: is improvement possible by parameter tweaking?

From left to right: Maximum Q value for action selection (averaged) Train error (averaged) Reward from environment (averaged) I run double Q-learning. A behavioral policy is ε-greedy, ε constant ...
3
votes
1answer
1k views

What is a minimal setup to solve the CartPole-v0 with DQN?

I solved the CartPole-v0 with a CEM agent pretty easily (experiments and code), but I struggle to find a setup which works with DQN. Do you know which parameters should be adjusted so that the mean ...
2
votes
0answers
37 views
+50

Actor-critic architecture: How is the policy updated?

I am going through the ddpg baseline code to try and gain an intuitive understanding of how the actor and critic networks function. DDPG has two components: the actor which is the deterministic ...
2
votes
0answers
16 views

Is it beneficial to use the last $N$ data points to train an RL agent?

Given that an environment in reinforcement learning is a Markov Decision Process (MDP), are there ever any cases where it is beneficial (or indeed where it makes sense) to use the last $N>1$ data ...
2
votes
0answers
22 views

Which one of these is the most efficient way to model training data for a neural network that will play a snake-like game?

I am building an AI using a neural network that will play Tron against a human player. The game consists of a board with fixed width and height where each player can move at any direction (except for ...
2
votes
0answers
33 views

Policy Gradient methods not converging to useful mean values

I am getting familiar with Policy Gradient methods, specifically Advantage Actor Critic (A2C). My target problem use clipped continuous state and action spaces and I have therefore been training my ...
2
votes
0answers
70 views

Optimal implementation of vanilla DQN loss in Keras

I've implemented vanilla DQN for continuous/non-images (no CNN) states in keras. But, I'm not sure if my implementation of the loss computation is optimal. For reminder the loss is defined as : $loss=...
2
votes
0answers
73 views

Reinforcement Learning using PPO2 in openai gym retro, mario not learning the clear the easy episode

I am training mario game in retro using ppo2 baselines for some time. I have tried level3 and level1 too. But even after full training when I play using saved checkpoints, the mario is not able to ...
2
votes
1answer
342 views

DQN fails to find optimal policy

Based on DeepMind publication, I've recreated the environment and I am trying to make the DQN find and converge to an optimal policy. The task of an agent is to learn how to sustainably collect apples ...
2
votes
0answers
387 views

Deep Reinforcement Learning for dynamic pricing

I am trying to implement a Deep Q Network model for Dynamic pricing in Logistics. I can define State Space (Origin, Destination, type of the shipment, customer, Type of the product, Commodity of the ...
2
votes
2answers
77 views

Puterman or Sutton Barto?

I wonder which of these two books is better to read for a beginner in RL and which are the pros and cons of them. Also, if you know any book that in your opinion is better for a beginner in RL, feel ...
2
votes
1answer
93 views

Reinforcement Learning on real time data over a web server

Question: is it possible to implement a reinforcement learning model over a NodeJS server? This server would be receiving binary forms of data (open /close; yes/no) in real time. The objective for ...
2
votes
0answers
316 views

Pytorch: How to create an update rule the doesn't come from derivatives?

I want to implement the following algorithm, taken from this book, section 13.6: Here, the neural networks' outputs are $V(S, w)$ and $\pi(A|S,\theta)$, parameterized by $w$ and $\theta$ respectively....
2
votes
1answer
63 views

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )? I want to replace the DNN section of Qlearning with a RF or SVR but the problem is that there is no clear ...
2
votes
0answers
381 views

Defining State Representation in Deep Q-Learning

So I am having difficulty difficulty figuring out exactly how I want to represent my environment state in my Deep Q-learning problem. Premise: There is a 2D grid space of which an agent needs to ...
2
votes
0answers
780 views

Initial Q-values in Q-Learning

I am running a Q-learning algorithm with a finite time horizon. Are 'optimistic initial conditions' still preferred if there is a possibility that some states will not be visited multiple times? ...
1
vote
0answers
9 views

Rewards are converged but with a lot of variations

I am training a reinforcement learning agent on an episodic task of fixed episode length. I am tracking the training process by plotting the cumulative rewards over an episode. I am using tensorboard ...
1
vote
0answers
10 views

Deep Q Learning - training slows down significantly

I'm trying to build a deep Q network to play snake. I designed the game so that the window is 600 by 600 and the snake's head moves 30 pixels each tick. I implemented the DQN algorithm with memory ...
1
vote
0answers
9 views

Autonomous learning - chatbots

My chatbots need to be trained when we get new data or feedbacks from users. Can someone provides ways how these chatbots can learn on themselves and become intelligent day by day? Some of techniques ...
1
vote
1answer
17 views

Training a model that has both 2D and 1D features using a CNN

I'm looking to pre-train a model for an RL agent but I'm having some trouble figuring some stuff out. Dataset: Minerl MineRLNavigateDense-v0 The observation space includes : 2D screen input (64,64) ...
1
vote
0answers
19 views

How can I increase the speed and performance of my implementation of an AI for Reversi?

I made an AI for Reversi, aka Othello (8×8), like Alpha Zero, using this book. This book is written in Japanese. The source code of the AI I implemented can be found in this Github repository. There ...
1
vote
0answers
18 views

DQN vs PG - when to use which?

I'd like to understand when using PG methods is more adequate than using DQN methods. Just to give a bit of background: I am currently using both APEX and R2D2 for my projects. Both work very well in ...
1
vote
0answers
12 views

What are regret bounds?

I searched for the term and it appears in a few articles but it is used without explanation. The only explanation I could find is in a PhD thesis: "Regret bounds are the common thread in the analysis ...
1
vote
2answers
102 views

Agent always takes a same action in DQN - Reinforcement Learning

I have trained an RL agent using DQN algorithm. After 20000 episodes my rewards are converged. Now when I test this agent, the agent is always taking the same action , irrespective of state. I find ...
1
vote
0answers
18 views

Policy Gradient custom loss function not working

I was experimenting with my policy gradient reinforcement learning algorithm, and I was wondering if I could use a similar method to the supervised cross-entropy. So, instead of using existing labels, ...
1
vote
0answers
18 views

How to teach algorithm to mimic paths in a certain enviroment

I have a set of scenarios which represent the movement of a car in a certain environment containing some obstacles. So for each scenario I have the position of the car (x,y,t) and a description of the ...
1
vote
1answer
75 views

How to avoid overfitting in Reinforcement Learning

I have implemented a RL model based on Deep Q-Learning for learning how to play a 2D game, like the ones in the OpenAI Gym. For testing the model, unlike most people, I have chosen to evaluate its ...
1
vote
0answers
12 views

temperature variable in boltzmmann-exploration in reinforcement learning

I have been using epsilon greedy action selection strategy and recently have come across boltzmann(softmax) action selection strategy. One thing I am not clear about boltzmann exploration is the ...
1
vote
0answers
66 views

boltzmann-exploration(softmax exploration) in reinforcement learning

I have started learning reinforcement learning and as a part of it I am exploring the action selection strategies available. I am comparing epsilon-greedy vs boltzmann exploration(softmax exploration)....
1
vote
0answers
21 views

Deep Q Learning - How is the ground truth obtained?

I am new to reinforcement learning so I apologize for the wrong use of terms, if any. In SARSA, the value of a state-action pair is updated after the robot takes an action following its internal ...
1
vote
0answers
25 views

Keras high loss and high accuracy in gk bot with reinforcement learning?

I'm making goal-keeper bot in haxball game. It worked well when i trained less but i worked worse when i trained more. Last reinforcement state: 5160 episode - 4171281 steps - 0.05 epsilon: Last fit ...
1
vote
0answers
28 views

How can I improve the performance of my DQN?

I created a deep Q network to play snake. The code works fine, except for the fact that performance doesn't really improve over the training cycle. At the end, it's pretty much indistinguishable from ...
1
vote
1answer
34 views

Guidelines to debug REINFORCE-type algorithms?

I implemented a self-critical policy gradient (as described here), for text summarization. However, after training, the results are not as high as expected (actually lower than without RL...). I'm ...
1
vote
0answers
15 views

Learning curve goes down after converge?

I trained an agent with policy gradient and the learning curve goes down after converges for a little while. Wondering if this is overfitting or some other issues? Thank you very much!
1
vote
1answer
73 views

DQN - how is it possible to train separate outputs for each action?

I'm trying to implement a Deep Q Network, but I'm stuck on how you train a network to predict multiple action-values when you can only collect data on a single action. In the paper it recommends ...
1
vote
1answer
15 views

Why can't Policy Gradient Algorithm be seen as an Actor-Critic Method?

During the equation deducing in policy gradient algorithm(e.g., REINFORCE), we are actually using an expectancy of total reward, which we try to maximize. $$\overline{R_\theta}=E_{\tau\sim\pi_\theta}[...
1
vote
1answer
47 views

What is the best way Reinforcement learning, RNN or others to predict the best action we have to take to maximize sales?

I have a dataset composed of few features : customerId, actionDay1, SalesDay1, actionDay20, SalesDay20, actionDay30, SalesDay30 action can be : call email face ...
1
vote
1answer
36 views

Expected value in Bellman equation

I am reading "reinforcement learning - An introduction" by Sutton and Barto. At pag. 59, there is the Bellman equation for the state-value function $\begin{array}{ll} v_{\pi}(s) &= \mathbb{E}_{...
1
vote
2answers
118 views

Alternative approach for Q-Learning

I have a question related to an alterative Q-Learning approach. I'd like to know if this already exists and I am not aware of it, or it doesn't exist because there are theoretical problems behind it. ...
1
vote
0answers
23 views

DQNs for huge or continuous state spaces

Have there been occasions where DQN failed to deal with huge state spaces? Can you point out a research paper regarding it?
1
vote
0answers
63 views

CartPole v1 - Simple backprop with 1 hidden layer

I'm trying to solve the CartPole-v1 problem from OpenAI by using backprop on a one-layer neural network - while updating the model at every time step using State action values (Q(s,a)). I'm unable to ...
1
vote
0answers
33 views

Q-value estimate in neural episodic control

Disclaimer: I'm not that familiar with reinforcement learning, so I might lack some basic knowledge on that topic. I was reading the neural episodic control (NEC) paper by Pritzel et al., 2017 and I'...