Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

0
votes
0answers
13 views

Why is the reward fluctuating for Double Q-Learning?

I am trying to implement Double Q-Learning using neural networks from the Keras library. When I first tried Simple DQN, the graph of the reward was fluctuating a lot so, I implemented a Double DQN. ...
1
vote
0answers
9 views

my PPO implementation for Cartpole, is code review allowed here?

I implemented the clipped objective PPO-clip as explained here: https://spinningup.openai.com/en/latest/algorithms/ppo.html Basically I used a dummy actor network to find the new action probability ...
1
vote
0answers
15 views

Dynamic pricing models when there are a lot of products with low demand

For products with high demand and much stock, there are several algorithms that work for dynamic pricing. However, when selling many different products in the same category and demand for each one is ...
1
vote
1answer
53 views

What will be the policy if the state space is continuous in Reinforcement learning

I have started recently with reinforcement learning. I have few doubts regarding the policy of an agent when it comes to continuous space. From my understanding, policy tells the agent which action ...
4
votes
1answer
33 views

Large action space for deep reinforcement learning

I know that in normal Deep Reinforcement Learning(DRL) scenario, we learn a deep neural network to map current states to Q values. The number of the Q values (# of outputs of the neural network) is ...
1
vote
0answers
19 views

Looking for a proper reinforcement learning solution

I am looking for a proper reinforcement learning solution for the following problem: Suppose I have a pool of candidate functions f \in Pool(it's like ...
0
votes
0answers
29 views

tensorflow eager execution outputs only same values

I'm trying to convert my tensorflow code to tensorflow eager. The problem is the forward pass predicts only the same actions for different input values in eager mode. The normal tensorflow code with ...
0
votes
1answer
35 views

Q-Learning experience replay: how to feed the neural network?

I'm trying to replicate the DQN Atari experiment. Actually my DQN isn't performing well; checking another one's codes, I saw something about experience replay which I don't understand. First, when you ...
0
votes
1answer
24 views

Policy gradient/REINFORCE algorithm with RNN: why does this converge with SGM but not Adam?

I am working on training RNN model on caption generation with REINFORCE algorithm. I adopt self-critic strategy (see paper Self-critical Sequence Training for Image Captioning) to reduce the variance. ...
0
votes
0answers
13 views

Different algorithms categorized in reinforcement learning

(Originally asked at cross validated forum: https://stats.stackexchange.com/questions/401615/different-algorithms-categorized-in-reinforcement-learning) For some time I am going through reinforcement ...
1
vote
1answer
37 views

Use deep reinforcement learning with recursive actions?

Can we use recursive actions in deep reinforcement learning ? If yes how ? For example, in a reassignment problem, for a task i the neural networks returns a node m, and then there will be some ...
1
vote
0answers
39 views

REINFORCE algorithm with discounted rewards – where does gamma^t in the update come from?

I'm looking at Sutton & Barto's rendition of the REINFORCE algorithm (from their book here, pg. 328). I can't quite understand why there is $\gamma^t$ on the last line. They say: [..] in the ...
0
votes
0answers
15 views

Implementation of actor-critic model for MountainCar

I'm trying to build a model for the Mountain Car game, following this Actor-Critic code: https://github.com/nikhilbarhate99/Actor-Critic (However, in this case, it's discrete action space, while it's ...
2
votes
1answer
90 views

DQN fails to find optimal policy

Based on DeepMind publication, I've recreated the environment and I am trying to make the DQN find and converge to an optimal policy. The task of an agent is to learn how to sustainably collect apples ...
0
votes
0answers
12 views

Speeding up actor critic training

I'm simulating a very simple system, recommendation system, and I am running an actor-critic model to predict what item I should recommend next. The agent is learning and is doing just fine. However, ...
0
votes
0answers
12 views

Is reward accumulated during a play iteration when performing SARSA?

I've been having issue with getting my DQN to converge to a good solution for snake. Regardless of the different types of reward functions I've tried, it seems that the snake is indefinitely going ...
0
votes
0answers
12 views

Learn User interests from feedback

I'm trying to create a pairwise learning model where user will be provided with 2 items (say $A$ and $B$) from which the user will provide the preference order ($A$ over $B$ or vice versa). Let's say ...
3
votes
1answer
75 views

Game theory in Reinforcement Learning

In one of the recent blog post by Deepmind, they have used game theory in Alpha Star algorithm. Deep Mind Alpha-Star: Mastering this problem requires breakthroughs in several AI research challenges ...
1
vote
0answers
7 views

Maximization problem with time-varying choice variables

I am looking at building a reinforcement learning problem where the objective function is say, profit. Agents in this setup can make different choices, depending on the time of interaction with the ...
0
votes
0answers
13 views

Modeling a Neural Network for a Turn-based Strategy (TBS) game

I'm modeling a neural network for a turn-based strategy game that involves purchase items every round. After the purchase phase, a combat phase is automatically executed (the winner gets more gold and ...
0
votes
0answers
10 views

Is DQN independent from type of input distribution?Why?

Can we say: Since DQN is online learning, it is independent of type of input distribution?
1
vote
1answer
29 views

In calculating policy gradients, wouldn't longer trajectories have more weight according to the policy gradient formula?

In Sergey Levine's lecture on policy gradients (berkeley deep rl course), he show that policy gradient can be evaluated according to the formula In this formula, wouldn't longer trajectories get more ...
1
vote
1answer
20 views

What are some ways to train, test and render an RL autonomous drone in a simulator?

I completed the Udacity nanodegree in Deep Learning but found the final project to be extremely difficult since the only visual feedback were from plots via matplotlib (rotor speeds, x/y/z coordinates,...
0
votes
0answers
15 views

Openai Spaces for a modified environment

I have a 2-dimensional array of normalized data. I am using space = np.array([0,1,...366],[0,0.000001,.....1]) I need to fit this as an observation space in ...
0
votes
2answers
45 views

What are features for state-action pairs in RL?

I read this answer: What are features in the context of reinforcement learning? But it only describes features for the state only in the context of cartpole, ie. Cart Position, Cart Velocity, Pole ...
3
votes
2answers
86 views

RL: Collecting States (training data) in real-life. Must use fixed timestep?

I am using a Reinforcement Learning agent to play a 3D game, but have trouble with collecting the "current and next state" pairs. To decide what action to perform, the network must perform a ...
5
votes
2answers
50 views

Why a Random Reward in One-step Dynamics MDP?

I am reading the 2018 book by Sutton & Barto on Reinforcement Learning and I am wondering the benefit of defining the one-step dynamics of an MDP as $$ p(s',r|s,a) = Pr(S_{t+1},R_{t+1}|S_t=s, ...
2
votes
1answer
17 views

How to write out the definition of the value function for continous action and state space

In the book of Sutton and Barto (2018) Reinforcement Learning: An Introduction. The author defines the value function as. $$v_{\pi}(\boldsymbol{s})=\mathbb{E}_{\boldsymbol{a}\,\sim\, \pi}\left[\sum_{...
2
votes
0answers
51 views

Deep Reinforcement Learning for dynamic pricing

I am trying to implement a Deep Q Network model for Dynamic pricing in Logistics. I can define State Space (Origin, Destination, type of the shipment, customer, Type of the product, Commodity of the ...
1
vote
0answers
17 views

Reinforcement learning - generating a matrix of continuous values with varying size for test data generation

Currently, I am using RL A3C algorithm for test data generation, where for a set of 30 functions written in C (mostly basic algorithms like Prime number checks, triangle validity, etc.) I try to ...
2
votes
1answer
57 views

Why is Reward Engineering considered “bad practice” in RL?

Reward engineering is an important part of supervised learning: Coming up with features is difficult, time-consuming, requires expert knowledge. "Applied machine learning" is basically feature ...
1
vote
1answer
30 views

Reward function to avoid illegal actions, minimize legal action and learn to win - Reinforcement Learning

I'm currently implementing PPO for a game with the following characteristics: Observation space: 9x9x(>150) Action space: 144 In a given state, only a handful of actions (~1-10) are legal The state ...
0
votes
1answer
50 views

Application of Deep Reinforcement Learning

I'm new to deep learning, and especially to reinforcement learning. I would like to know if it's possible to predict which combination of hashtags (from a subset of chosen hashtags) would produce the ...
1
vote
0answers
15 views

Which reinforcement learning methods can be trained off-policy?

My understanding is that: Value-Based methods such as DQN, C51, Rainbow DQN can naturally be trained off-policy using a Replay Buffer, without having to account for any kind of off-policy correction. ...
2
votes
2answers
36 views

Puterman or Sutton Barto?

I wonder which of these two books is better to read for a beginner in RL and which are the pros and cons of them. Also, if you know any book that in your opinion is better for a beginner in RL, feel ...
2
votes
1answer
23 views

Reinforcement Learning on real time data over a web server

Question: is it possible to implement a reinforcement learning model over a NodeJS server? This server would be receiving binary forms of data (open /close; yes/no) in real time. The objective for ...
1
vote
1answer
28 views

Evaluating value functions in RL

I'm working my way through the book Reinforcement Learning by Richar S. Sutton and Andrew G. Barto and I am stuck on the following question. The value of a state depends on the the values of the ...
0
votes
1answer
25 views

Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

I am looking for a little clarity on what the policy gradient theorem means. My confusion lies in the fact that the reward $R$ in reinforcement learning is non-differentiable in the policy parameters. ...
1
vote
1answer
80 views

Off-policy n-step learning with DQN

I'm reviewing the Rainbow paper and I'm not sure I understand how they can use DQN with multi-step learning, without doing any correction to account for off-policiness. So. I understand how you can ...
2
votes
1answer
42 views

In Keras library, what is the meaning of “nb_steps_warmup” in the DQNAgent Object initialization?

I can't understand the meaning of "nb_steps_warmup", a parameter of the __init__ function of DQNAgent class of the Keras_RL module. I just know that when I set ...
1
vote
0answers
38 views

What strategies and algorithms are suited for using the time wasted in collecting big data?

Most state of art algorithms right now is using/exploiting big data. My concern is what can you do to maximize reward while waiting for large amount of data that you think is appropriate. For ...
1
vote
1answer
34 views

Problem when cherry picking actions - Proximal Policy Optimization

I'm using the implementation of PPO2 in stable-baselines (a fork of OpenAI's baselines) for a RL-problem. My observation space is 9x9x191 and my action space is 144. Given a state, only some actions ...
4
votes
1answer
44 views

how to choose between discounted reward and average reward

how to select between average reward and discounted reward? And when average reward is more effective in comparison with discounter reward and when vice versa is correct? -Is is possible to use both ...
2
votes
0answers
97 views

Pytorch: How to create an update rule the doesn't come from derivatives?

I want to implement the following algorithm, taken from this book, section 13.6: Here, the neural networks' outputs are $V(S, w)$ and $\pi(A|S,\theta)$, parameterized by $w$ and $\theta$ respectively....
1
vote
1answer
60 views

Hindsight experience replay: strategy for sampling goals

The authors of Hindsight Experience Replay list out several strategies for sampling a set of additional goals $G$ in Section 4.5: final - corresponds with final state of environment, future — replay ...
0
votes
0answers
18 views

Can we do convolutions on binary mask inputs?

I am training a vehicle trajectory prediction algorithm using Deep MaxEnt Inverse Reinforcement Learning (https://arxiv.org/abs/1507.04888). My intention is to have as input to this algorithm a top-...
4
votes
2answers
55 views

Is there a rule of thumb when designing neural network in deep reinforcement learning?

In deep learning, we can assess model's performance with loss function value and improve model's performance with K-fold cross-validation and so on. But how can we design and tune neural network used ...
0
votes
2answers
19 views

Proof subtracting baseline doesn't influence gradient can be used to show no gradient exist at all?

I am using David Silver's course in RL to help me write my thesis. However, I am baffled by the proof given in lecture 7 slide 29: slideshow \begin{align} \mathbb{E}_{\pi_\theta}[\nabla_\theta \log_\...
0
votes
0answers
21 views

Q-Learning where some actions are more difficult to predict than others

I'm trying to train a deep Q network to optimize play in a game. For simplicity, let's say my game only has two actions, A and B. The reward distribution for A is somewhat uniform in the range -1 to ...
0
votes
2answers
45 views

Gym action space for board game with reward function

Im trying to design an openai gym environment that plays a quite simple board game where each player has 16 pieces that are exactly the same in regard to how they can move. The board is 10x10 and ...