Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Filter by
Sorted by
Tagged with
3
votes
1answer
107 views

Game theory in Reinforcement Learning

In one of the recent blog post by Deepmind, they have used game theory in Alpha Star algorithm. Deep Mind Alpha-Star: Mastering this problem requires breakthroughs in several AI research challenges ...
1
vote
0answers
9 views

Maximization problem with time-varying choice variables

I am looking at building a reinforcement learning problem where the objective function is say, profit. Agents in this setup can make different choices, depending on the time of interaction with the ...
0
votes
0answers
51 views

Modeling a Neural Network for a Turn-based Strategy (TBS) game

I'm modeling a neural network for a turn-based strategy game that involves purchase items every round. After the purchase phase, a combat phase is automatically executed (the winner gets more gold and ...
0
votes
0answers
15 views

Is DQN independent from type of input distribution?Why?

Can we say: Since DQN is online learning, it is independent of type of input distribution?
1
vote
1answer
44 views

In calculating policy gradients, wouldn't longer trajectories have more weight according to the policy gradient formula?

In Sergey Levine's lecture on policy gradients (berkeley deep rl course), he show that policy gradient can be evaluated according to the formula In this formula, wouldn't longer trajectories get more ...
1
vote
1answer
83 views

What are some ways to train, test and render an RL autonomous drone in a simulator?

I completed the Udacity nanodegree in Deep Learning but found the final project to be extremely difficult since the only visual feedback were from plots via matplotlib (rotor speeds, x/y/z coordinates,...
0
votes
1answer
57 views

Openai Spaces for a modified environment

I have a 2-dimensional array of normalized data. I am using space = np.array([0,1,...366],[0,0.000001,.....1]) I need to fit this as an observation space in ...
0
votes
2answers
351 views

What are features for state-action pairs in RL?

I read this answer: What are features in the context of reinforcement learning? But it only describes features for the state only in the context of cartpole, ie. Cart Position, Cart Velocity, Pole ...
3
votes
2answers
99 views

RL: Collecting States (training data) in real-life. Must use fixed timestep?

I am using a Reinforcement Learning agent to play a 3D game, but have trouble with collecting the "current and next state" pairs. To decide what action to perform, the network must perform a ...
5
votes
2answers
100 views

Why a Random Reward in One-step Dynamics MDP?

I am reading the 2018 book by Sutton & Barto on Reinforcement Learning and I am wondering the benefit of defining the one-step dynamics of an MDP as $$ p(s',r|s,a) = Pr(S_{t+1},R_{t+1}|S_t=s, ...
2
votes
1answer
20 views

How to write out the definition of the value function for continous action and state space

In the book of Sutton and Barto (2018) Reinforcement Learning: An Introduction. The author defines the value function as. $$v_{\pi}(\boldsymbol{s})=\mathbb{E}_{\boldsymbol{a}\,\sim\, \pi}\left[\sum_{...
2
votes
0answers
388 views

Deep Reinforcement Learning for dynamic pricing

I am trying to implement a Deep Q Network model for Dynamic pricing in Logistics. I can define State Space (Origin, Destination, type of the shipment, customer, Type of the product, Commodity of the ...
1
vote
0answers
25 views

Reinforcement learning - generating a matrix of continuous values with varying size for test data generation

Currently, I am using RL A3C algorithm for test data generation, where for a set of 30 functions written in C (mostly basic algorithms like Prime number checks, triangle validity, etc.) I try to ...
2
votes
1answer
153 views

Why is Reward Engineering considered “bad practice” in RL?

Reward engineering is an important part of supervised learning: Coming up with features is difficult, time-consuming, requires expert knowledge. "Applied machine learning" is basically feature ...
1
vote
2answers
82 views

Reward function to avoid illegal actions, minimize legal action and learn to win - Reinforcement Learning

I'm currently implementing PPO for a game with the following characteristics: Observation space: 9x9x(>150) Action space: 144 In a given state, only a handful of actions (~1-10) are legal The state ...
0
votes
1answer
67 views

Application of Deep Reinforcement Learning

I'm new to deep learning, and especially to reinforcement learning. I would like to know if it's possible to predict which combination of hashtags (from a subset of chosen hashtags) would produce the ...
1
vote
0answers
20 views

Which reinforcement learning methods can be trained off-policy?

My understanding is that: Value-Based methods such as DQN, C51, Rainbow DQN can naturally be trained off-policy using a Replay Buffer, without having to account for any kind of off-policy correction. ...
2
votes
2answers
77 views

Puterman or Sutton Barto?

I wonder which of these two books is better to read for a beginner in RL and which are the pros and cons of them. Also, if you know any book that in your opinion is better for a beginner in RL, feel ...
2
votes
1answer
93 views

Reinforcement Learning on real time data over a web server

Question: is it possible to implement a reinforcement learning model over a NodeJS server? This server would be receiving binary forms of data (open /close; yes/no) in real time. The objective for ...
1
vote
1answer
38 views

Evaluating value functions in RL

I'm working my way through the book Reinforcement Learning by Richar S. Sutton and Andrew G. Barto and I am stuck on the following question. The value of a state depends on the the values of the ...
0
votes
1answer
55 views

Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

I am looking for a little clarity on what the policy gradient theorem means. My confusion lies in the fact that the reward $R$ in reinforcement learning is non-differentiable in the policy parameters. ...
2
votes
1answer
500 views

Off-policy n-step learning with DQN

I'm reviewing the Rainbow paper and I'm not sure I understand how they can use DQN with multi-step learning, without doing any correction to account for off-policiness. So. I understand how you can ...
2
votes
1answer
408 views

In Keras library, what is the meaning of “nb_steps_warmup” in the DQNAgent Object initialization?

I can't understand the meaning of "nb_steps_warmup", a parameter of the __init__ function of DQNAgent class of the Keras_RL module. I just know that when I set ...
1
vote
0answers
41 views

What strategies and algorithms are suited for using the time wasted in collecting big data?

Most state of art algorithms right now is using/exploiting big data. My concern is what can you do to maximize reward while waiting for large amount of data that you think is appropriate. For ...
1
vote
1answer
132 views

Problem when cherry picking actions - Proximal Policy Optimization

I'm using the implementation of PPO2 in stable-baselines (a fork of OpenAI's baselines) for a RL-problem. My observation space is 9x9x191 and my action space is 144. Given a state, only some actions ...
5
votes
1answer
235 views

how to choose between discounted reward and average reward

how to select between average reward and discounted reward? And when average reward is more effective in comparison with discounter reward and when vice versa is correct? -Is is possible to use both ...
2
votes
0answers
317 views

Pytorch: How to create an update rule the doesn't come from derivatives?

I want to implement the following algorithm, taken from this book, section 13.6: Here, the neural networks' outputs are $V(S, w)$ and $\pi(A|S,\theta)$, parameterized by $w$ and $\theta$ respectively....
1
vote
1answer
250 views

Hindsight experience replay: strategy for sampling goals

The authors of Hindsight Experience Replay list out several strategies for sampling a set of additional goals $G$ in Section 4.5: final - corresponds with final state of environment, future — replay ...
0
votes
0answers
36 views

Can we do convolutions on binary mask inputs?

I am training a vehicle trajectory prediction algorithm using Deep MaxEnt Inverse Reinforcement Learning (https://arxiv.org/abs/1507.04888). My intention is to have as input to this algorithm a top-...
4
votes
2answers
83 views

Is there a rule of thumb when designing neural network in deep reinforcement learning?

In deep learning, we can assess model's performance with loss function value and improve model's performance with K-fold cross-validation and so on. But how can we design and tune neural network used ...
0
votes
2answers
33 views

Proof subtracting baseline doesn't influence gradient can be used to show no gradient exist at all?

I am using David Silver's course in RL to help me write my thesis. However, I am baffled by the proof given in lecture 7 slide 29: slideshow \begin{align} \mathbb{E}_{\pi_\theta}[\nabla_\theta \log_\...
0
votes
0answers
23 views

Q-Learning where some actions are more difficult to predict than others

I'm trying to train a deep Q network to optimize play in a game. For simplicity, let's say my game only has two actions, A and B. The reward distribution for A is somewhat uniform in the range -1 to ...
0
votes
2answers
154 views

Gym action space for board game with reward function

Im trying to design an openai gym environment that plays a quite simple board game where each player has 16 pieces that are exactly the same in regard to how they can move. The board is 10x10 and ...
0
votes
0answers
19 views

Is DQN a SMART (Semi Markov Average Reward Technique)?

Is DQN a SMART (Semi Markov Average Reward Technique) algorithm?
0
votes
0answers
48 views

About applying time series forecasting to problems better suited for reinforcement learning, like toy example “Jack's car rental”

"Jack's car rental" is an example of a reinforcement learning problem, proposed in the Sutton & Barto book, in which the goal is to optimize the daily distribution of cars in two locations of the ...
0
votes
1answer
46 views

Why is “next state” kept in RL experience replay?

Following this explanation on what is experience replay (and others), I noticed an experience element is defined as $e_t = (s_t,a_t,r_t,s_{t+1})$ My question is, why do we need the ...
1
vote
0answers
24 views

What is the meaning of the Variant Q-learning and To what INPUT and OUTPUT refer? in Abstract of DeepMind DQN paper 2013

-INPUT and OUTPUT OF ATARI DQN: In the abstract paragraph of the DQN work by DeepMind https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf it has written: " We present the first deep learning model to ...
2
votes
1answer
679 views

What is the difference between dynamic programming and Q-learning?

What is the difference between the DP-based algorithm and Q-learning?
0
votes
0answers
54 views

Proxy task for CNN architecture search

I would like to explore possible model's architectures to improve the object detection accuracy. The hyper-parameters I would like to optimize are the number of filters for each convolution layer and ...
1
vote
0answers
68 views

In DQN, what is the real reason we don't backpropagate through the target network?

Actually, if we are to backpropagate through the target network, there is no use for the target network anymore. Let's say we don't use any target network. We hence enforce only the "temporal ...
0
votes
0answers
22 views

How apply Reinforcement Learning in the following case?

Suppose that I have to move from point A to B and I have to choose among 3 different paths. But we don't know the traffic in each path, so what is the training rule to use to learn the best behaviour? ...
0
votes
1answer
642 views

RL - Weighthing negative rewards [closed]

Let's consider that I give an agent a reward of -1 (minimum reward) every time it performs an action which leads to the premature end of the episode (i.e., the agent dies). Besides, I also give a ...
2
votes
1answer
3k views

Reinforcement learning for continuous state and action space

Problem My goal is to apply Reinforcement Learning to predict the next state of an object under a known force in a 3D environment (the approach would be reduced to supervised learning, off-line ...
0
votes
1answer
63 views

How to represent an image as state in a Q-table

I'm trying to do Q-learning with the Atari games using the gym python's package. I want to use the image as the state of my algorithm, but I came up with a doubt: ...
0
votes
1answer
48 views

How does Q-Learning deal with mixed strategies?

I'm trying to understand how Q-learning deals with games where the optimal policy is a mixed strategy. The Bellman equation says that you should choose $max_a(Q(s,a))$ but this implies a single unique ...
1
vote
1answer
69 views

Why not use max(returns) instead of average(returns) in off-policy Monte Carlo control?

As I understand it, in reinforcement learning, off-policy Monte Carlo control is when the state-action value function $Q(s,a)$ is estimated as a weighted average of the observed returns. However, in ...
3
votes
1answer
2k views

Can Reinforcement learning be applied in image classification?

So my question is can Reinforcement learning be applied in image classification?
1
vote
1answer
38 views

In Reinforcement Learning can I randomly assign next_states from the state space to my agent while creating transition set?

In Reinforcement Learning, while creating transition samples (state, action, next_state, reward), where: Agent: The learning agent Environment: The trainer The environment gives two feedback to the ...
1
vote
1answer
252 views

Reinforcement learning: negative reward (punish) illegal actions?

If you train an agent using reinforcement learning (with Q-function in this case), should you give a negative reward (punish) if the agent proposes illegal actions for the presented state? I guess ...
1
vote
1answer
82 views

Policy gradient - and auto-differentiation (Pytorch/Tensorflow)

In policy gradient, we have something like this: Is my understanding correct that if I apply log cross-entropy on the last layer, the gradient will be automatically calculated as per formula above?