Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

216 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8
votes
0answers
172 views

Formal proof of vanilla policy gradient convergence

So I stumbled upon this question, where the author asks for a proof of vanilla policy gradient procedures. The answer provided points to some literature, but the formal proof is nowhere to be included....
4
votes
1answer
31 views

Reinforcement learning with sparse acting agent

I'm working on a problem where the optimal policy involves the agent "doing nothing" most of the time, and "doing something" during rare critical moments. Is there any literature or best practices ...
4
votes
2answers
93 views

Is there a rule of thumb when designing neural network in deep reinforcement learning?

In deep learning, we can assess model's performance with loss function value and improve model's performance with K-fold cross-validation and so on. But how can we design and tune neural network used ...
3
votes
0answers
36 views

NN training with repetitive features

I posted the question also on ai.stackexchange but it didn't get any answers so I though I could try here. Here is a copy paste: Let's say you are training a NN in a RL setting where the state (i.e. ...
3
votes
0answers
598 views

Deep Reinforcement Learning for dynamic pricing

I am trying to implement a Deep Q Network model for Dynamic pricing in Logistics. I can define State Space (Origin, Destination, type of the shipment, customer, Type of the product, Commodity of the ...
3
votes
1answer
222 views

Time horizon T in policy gradients (actor-critic)

I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture. At the bottom of that slide, the gradient of the expected sum of rewards ...
3
votes
0answers
105 views

Deep advantage learning: how to predict the value

I'm currently working on a collection of reinforcement algorithms: https://github.com/lhk/rl_gym For deep q-learning, you need to calculate the q-values that should be predicted by your network. ...
3
votes
1answer
127 views

Choosing a right algorithm for template-based text generation

I am doing a text generation project -- the task is to basically represent the statistical data in a readable way. The way I decided to go about this is template-based: each data type has a template ...
3
votes
0answers
136 views

Graphical results of Q-Learning: is improvement possible by parameter tweaking?

From left to right: Maximum Q value for action selection (averaged) Train error (averaged) Reward from environment (averaged) I run double Q-learning. A behavioral policy is ε-greedy, ε constant ...
2
votes
0answers
14 views

Bandit program using epsilon-greedy

I started to learn reinforcement learning, the first example is handling bandit program using epsilon-greedy method, In this example, there are three bandit machines used, the output is the mean value ...
2
votes
0answers
20 views

What's the definition of retraining?

In transfer learning, we always use new data to retrain the pre-trained model. But, what is the specific and official definition of retraining? Or what papers mentioned this definition, in transfer ...
2
votes
0answers
18 views

Markov Decision Process representation

I'm attempting to model a simple process using a Markov Decision Process. Let $A$ be a set of $3$ actions : $ A \in \{b,s\}$. $T(s,a,s')$ represents the probability of if in state $s$ , take action $...
2
votes
0answers
28 views

Actions taken by agentn/ agent performance not improving

Hi I am trying to develop an rl agent using PPO algorithm. My agent takes an action(CFM) to maintain a state variable called RAT in between 24 to 24.5. I am using PPO algorithm of stable-baselines ...
2
votes
0answers
30 views

Incentivizing curiosity in a sparse reward environment

I'm quite new to reinforcement learning, but have been exploring different kinds of architectures (DQN, dueling DQN, actor critic, etc.) and evaluating their ability to solve certain problems. The ...
2
votes
1answer
24 views

reinforcement learning: Decompose a policy gradient

I am studying the policy gradient through the website: https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f Couldn't figure out how the first equation becomes the second ...
2
votes
0answers
16 views

Is it beneficial to use the last $N$ data points to train an RL agent?

Given that an environment in reinforcement learning is a Markov Decision Process (MDP), are there ever any cases where it is beneficial (or indeed where it makes sense) to use the last $N>1$ data ...
2
votes
0answers
22 views

Which one of these is the most efficient way to model training data for a neural network that will play a snake-like game?

I am building an AI using a neural network that will play Tron against a human player. The game consists of a board with fixed width and height where each player can move at any direction (except for ...
2
votes
0answers
111 views

Policy Gradient methods not converging to useful mean values

I am getting familiar with Policy Gradient methods, specifically Advantage Actor Critic (A2C). My target problem use clipped continuous state and action spaces and I have therefore been training my ...
2
votes
0answers
156 views

Optimal implementation of vanilla DQN loss in Keras

I've implemented vanilla DQN for continuous/non-images (no CNN) states in keras. But, I'm not sure if my implementation of the loss computation is optimal. For reminder the loss is defined as : $loss=...
2
votes
0answers
100 views

Reinforcement Learning using PPO2 in openai gym retro, mario not learning the clear the easy episode

I am training mario game in retro using ppo2 baselines for some time. I have tried level3 and level1 too. But even after full training when I play using saved checkpoints, the mario is not able to ...
2
votes
1answer
515 views

DQN fails to find optimal policy

Based on DeepMind publication, I've recreated the environment and I am trying to make the DQN find and converge to an optimal policy. The task of an agent is to learn how to sustainably collect apples ...
2
votes
2answers
128 views

Puterman or Sutton Barto?

I wonder which of these two books is better to read for a beginner in RL and which are the pros and cons of them. Also, if you know any book that in your opinion is better for a beginner in RL, feel ...
2
votes
1answer
115 views

Reinforcement Learning on real time data over a web server

Question: is it possible to implement a reinforcement learning model over a NodeJS server? This server would be receiving binary forms of data (open /close; yes/no) in real time. The objective for ...
2
votes
0answers
450 views

Pytorch: How to create an update rule the doesn't come from derivatives?

I want to implement the following algorithm, taken from this book, section 13.6: Here, the neural networks' outputs are $V(S, w)$ and $\pi(A|S,\theta)$, parameterized by $w$ and $\theta$ respectively....
2
votes
1answer
89 views

How does Q-Learning deal with mixed strategies?

I'm trying to understand how Q-learning deals with games where the optimal policy is a mixed strategy. The Bellman equation says that you should choose $max_a(Q(s,a))$ but this implies a single unique ...
2
votes
1answer
2k views

IndexError: index 804 is out of bounds for axis 0 with size 800

i installed a self driving car project from superdatascience site , when i open the map using terminal after a while the map window close up or it closes directly after i maximize the map window and ...
2
votes
1answer
38 views

objective in policy gradient equation?

I don't understand how this was deduced from first equation to second expectation. Is it from conditional probability theory? I checked but still can't understand. From wikipedia, the expectation of a ...
2
votes
1answer
69 views

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )? I want to replace the DNN section of Qlearning with a RF or SVR but the problem is that there is no clear ...
2
votes
0answers
553 views

Defining State Representation in Deep Q-Learning

So I am having difficulty difficulty figuring out exactly how I want to represent my environment state in my Deep Q-learning problem. Premise: There is a 2D grid space of which an agent needs to ...
2
votes
0answers
169 views

Epoch greedy algorithm for contextual bandits

I'm reading the following paper on the epoch greedy algorithm for the contextual bandits problem. I have two questions http://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf I'm unsure ...
2
votes
0answers
875 views

Initial Q-values in Q-Learning

I am running a Q-learning algorithm with a finite time horizon. Are 'optimistic initial conditions' still preferred if there is a possibility that some states will not be visited multiple times? ...
1
vote
0answers
13 views

how do deep Q network deal with varying input size?

I am conducting research with multiply agents in an environment. The main concept of my methodology is a centralized control system, which means we take the positions, as well as other information, of ...
1
vote
0answers
6 views

Caps_Net. searching for example and library to use

Which library is most recommended and easy to use?
1
vote
0answers
114 views

Pytorch XLA to solve the spawn problems in a Colab Env

As reference only, here is my code It seems that torch.multiprocessing.set_start_method("spawn") can't be used in an Colab Env. Only 'fork' is allowed. I have ...
1
vote
1answer
11 views

Parameter initialization in a genetic algorithm

I'm using a neural network in a genetic algorithm. The neural network has 4 inputs (values between 0 and 1) and 4 outputs, corresponding to the probabilities of different actions. The neural network ...
1
vote
0answers
59 views

Is this an implementation of Reinforcement learning?

I've written code in Python to replicate the results of a brief and simple paper about reinforcement learning. A brief description of the problem: we have a generated time series of returns (that ...
1
vote
0answers
12 views

Recommendation system with active learning

I have data where companies ask users to score a bunch of questions but some items may be randomly chosen while others are personalized. Users score higher in personalized questions on average. I have ...
1
vote
0answers
13 views

Informed sender in Lazaridou et al. (2017) 's guessing game

I'm trying to implement the informed sender from Multi-Agent Cooperation and the Emergence of (Natural) Language (Lazaridou et al., 2017). However I'm confused about the tensor shape in the forward ...
1
vote
0answers
22 views

Definition of obstacles in new OpenAI gym environment

I study a Reinforcement Learning algorithm that navigate an agent from one initial point to another in a complex environment where other agents and obstacles exists too. I want to make my own gym ...
1
vote
0answers
51 views

Sudden drop of score in the last few episodes

I was following this tutorial about lunar lander and deep Q learning with Tensorflow 2 and I noticed something odd. The problem was actually solved at episode 476 but then the score went from 259.90 ...
1
vote
0answers
58 views

Reward(t) vs. Reward(t+1) ? Reinforcement Learning, Q-learning

In "Reinforcement Learning, An Introduction; Richard S. Sutton and Andrew G. Barto" is written on page 70: 3.1 The Agent–Environment Interface At each time step t, the agent receives some ...
1
vote
0answers
85 views

Q-learning, state transition, immediate rewards (trading logic)

I've been thinking about how to correctly calculate rewards for several weeks now. Here is a grid example: ...
1
vote
0answers
101 views

Deep reinforcement learning with multi-dimensional action

I am trying to design reinforcement learning algorithm. My action and state space are continuous. Action, which I would like to take can be represented by a matrix, lets say of dimension $n \times n$. ...
1
vote
0answers
29 views

Cold-start problem in Real Time Bidding

I'm currently on the reading stage of the deployment of an RTB system. I've seen the problem of a cold start (having no initial guess of how to bid) in several papers, but I haven't really seen it be ...
1
vote
2answers
32 views

Idenitity between TD(0) algorithm and Policy Evaluation in Dynamic Programming when alpha is equal to 1

TD(0) algorithm is defined as the iterative update of the following: $$ V(s) \leftarrow V(s) + \alpha({r + \gamma V(s')} - V(s) ) $$ Now, if we assume alpha to be equal to 1, we get the traditional ...
1
vote
1answer
34 views

How to improve tensorflow 2.0 code for policy gradient?

I recreated some code I found online for solving the bandits problem using policy gradient. The example was in tensorflow 1.0 so I recreated it with tensorflow 2.0 using eager execution and gradient ...
1
vote
0answers
93 views

Why is orthogonal weights initialization so important for PPO?

I have implemented PPO to solve Atari environments. For the longest time I couldn't figure out why my model would not converge as fast as other open source solutions. Eventually it boiled down to this ...
1
vote
0answers
17 views

Reinforcement learning converges for mean loss but not for each training data

Here I show a dummy example that represents my actual problem. My neural network (NN) receives one input and gives the probabilities for two output nodes. The code for the NN is: ...
1
vote
0answers
304 views

reinforcement learning: PPO vs. DDPG vs. TRPO - difference and intuition

I know there is a lot of blog talk about the PPO, DDPG and TRPO, but I am wondering would it be possible to explain the differences of these methods in layman's term? What's the intuition behind them ...
1
vote
0answers
20 views

Rewards are converged but with a lot of variations

I am training a reinforcement learning agent on an episodic task of fixed episode length. I am tracking the training process by plotting the cumulative rewards over an episode. I am using tensorboard ...

1
2 3 4 5