Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

How is the poicy gradient's cost function and gradients work?

I am not a math expert but have a basic understanding of linear algebra,calculus and probability and understands the math behind backprop. Currently I am trying to learn about policy gradient ...
0
votes
0answers
15 views

Reducing the training time of an RL agent

I am trying to develop an rl agent using DQN algorithm.During training, the agent interacts with environment which is a simulated one.Each episode takes around 10 mins to run. This way if want my ...
0
votes
0answers
11 views

Prioritized Experience Replay - for whole episodes

I want to use Prioritized Experience Replay for whole episodes, instead for single transitions. What's the best way to define the priorities as episodes can be of different lengths? Personally I can ...
1
vote
1answer
19 views

How to formulate reward of an rl agent with two objectives

I have started learning reinforcement learning and trying to apply it for my use case. I am developing an rl agent which can maintain temperature at a particular value, and minimize the energy ...
1
vote
1answer
32 views

Policy gradient vs cost function

I was working with continuous system RL and obviously stumbled across this Policy Gradient. I want to know is this something like cost function for RL? It kinda gives that impression considering we ...
1
vote
0answers
13 views

Deep Q Learning - How is the ground truth obtained?

I am new to reinforcement learning so I apologize for the wrong use of terms, if any. In SARSA, the value of a state-action pair is updated after the robot takes an action following its internal ...
0
votes
0answers
18 views

How can Reinforcement Learning Work in Trading?

I have recently become interested in Reinforcement Learning, mostly as a result of Alpha Zero’s success in the chess (and my own enthusiasm for the game). While I understand the utility of RL in board ...
2
votes
1answer
23 views

Modelling of an environment that is stochastic in nature

I have started learning reinforcement learning and have few doubts regarding model based and model free methods. Is it possible to model an environment that is stochastic in nature? Is it because ...
1
vote
0answers
15 views

Keras high loss and high accuracy in gk bot with reinforcement learning?

I'm making goal-keeper bot in haxball game. It worked well when i trained less but i worked worse when i trained more. Last reinforcement state: 5160 episode - 4171281 steps - 0.05 epsilon: Last fit ...
0
votes
0answers
7 views

How can I create a loss function in Keras for policy gradient that deals with the fact episodes have varying length?

I am implementing policy gradient to solve the OpenAI CartPole game. I have a loss function that goes as follows: ...
0
votes
0answers
13 views

Why residual gradient algorithm is stable to converge to a suboptimal?

naive residual-algorithm discribed in book RLAI Chapter 11.5 by Sutton and Barto is worked as: $$ \begin{aligned} \mathbf{w}_{t+...
0
votes
0answers
10 views

Hints for large, variable-size action space

I am making an environment using OpenAI gym for Diplomacy, and making an AI for it. In Diplomacy, a player has many units, and each unit has a number of moves available to it. Therefore, the player'...
1
vote
1answer
18 views

Is DQN limited to working with only image frames?

I have few questions about Deep Q Network. Does DQN only accept image frames as input? I have never hear (read) a paperwork where it doesn't use image frames. If the first is a No, then does image ...
1
vote
0answers
27 views

How can I improve the performance of my DQN?

I created a deep Q network to play snake. The code works fine, except for the fact that performance doesn't really improve over the training cycle. At the end, it's pretty much indistinguishable from ...
0
votes
0answers
10 views

Discontinuous jumps in reward as a function of episode for deep reinforcement learning?

I'm using deep deterministic policy gradient (DDPG) with an Ornstein Uhlenbeck process as exploration noise. The agent manages to achieve its goal in a suitable manner, so I'm happy about the ...
0
votes
0answers
14 views

How should I interpret the weights file of the Leela Zero neural network?

I am trying to understand the NN architecture given at https://github.com/leela-zero/leela-zero/blob/next/training/caffe/zero.prototxt. So, I downloaded the NN weights from http://zero.sjeng.org/. ...
1
vote
1answer
25 views

Why DDPG's policy target is Q-value itself?

Could someone explain why the target of the DDPG's policy is $Q(s,\mu(s))$? My understanding of DDPG is like this. Since it is intractable to calculate the $argmax_a Q(s,a)$ in a continuous space, ...
0
votes
0answers
20 views

toy trading bot - deep reinforcement learning

im trying to apply RL to options trading and i guess problem formulation is the hardest part here. Given below is the formulation and chunks of code supporting it. Kindly review and lmk if there's ...
0
votes
1answer
13 views

Unbalanced discounted reward in reinforcement learning : is it a problem?

Discounted rewards seems unbalanced to me. If we take as example an episode with 4 actions, where each action receive a reward of +1 : +1 -> +1 -> +1 -> +1 The discounted reward for the last ...
0
votes
1answer
26 views

Understanding REINFORCE loss

The loss used in REINFORCE algorithm is confusing me. From Pytorch documentation : loss = -m.log_prob(action) * reward We want to minimize this loss. If a ...
0
votes
0answers
24 views

Are convolutional layers necessary for deep Q networks?

I'm currently trying to build a deep Q network to play the classic Snake game. I designed the game in such a way that the state space is confined to a 20 x 20 matrix, with 1's representing a square ...
1
vote
1answer
26 views

Guidelines to debug REINFORCE-type algorithms?

I implemented a self-critical policy gradient (as described here), for text summarization. However, after training, the results are not as high as expected (actually lower than without RL...). I'm ...
0
votes
1answer
15 views

RL Sutton book, initial estimate of q*(a) for 10 arm testbed

The Sutton book does not mention what the initial estimate is for q*(a) before the first reward is received. In this code repo that seems to go along with the book: Sutton code repo They have ...
0
votes
0answers
16 views

How to model this sequential decision-making problem?

There is a gambling problem in which every 10 seconds, the player should decide on the amount of bet ($b$) and a factor ($\alpha_{1}$). Then a new factor generated by the system ($\alpha_{2}$) and is ...
3
votes
1answer
120 views

Is RL applied to animal dispersion a valid approach?

I have an agent which has a medium-sized, discrete set of actions $A$: $10<|A|<100$. The actions can be taken over an infinite horizon of 1 second per timestep $t$. The world is essentially ...
1
vote
0answers
14 views

Learning curve goes down after converge?

I trained an agent with policy gradient and the learning curve goes down after converges for a little while. Wondering if this is overfitting or some other issues? Thank you very much!
1
vote
1answer
35 views

Q-learning when minimising a total cost instead of maximising a total reward

I have a decision problem where the results are measured as a cost that I want to minimise. It seems like a good fit to Q-learning, but I am not sure how to adjust it to deal with a cost instead of a ...
0
votes
1answer
25 views

Does an RL agent still learn if its actions are “blocked”?

Say we have a game that is a maze environment where there is a character to be controlled through the maze. When the agent (the character) approaches a wall, it may try to execute an action that would ...
0
votes
1answer
34 views

Deep Q-Learning for physical quantity: q-values distribution not as expected

Setting I am trying to learn a specific physical quantity (radiance) inside a 3D scene with Deep Q-Learning. Just to give a quick overview, my agent shoots rays inside the scene: the reward is the ...
2
votes
0answers
16 views

Is it beneficial to use the last $N$ data points to train an RL agent?

Given that an environment in reinforcement learning is a Markov Decision Process (MDP), are there ever any cases where it is beneficial (or indeed where it makes sense) to use the last $N>1$ data ...
2
votes
1answer
52 views

Why do trained RL agents still display stochastic “exploratory” behavior on testing data?

I am training a PPO2 RL model using stable baselines. One thing I found is that a trained agent will still display some stochastic behavior on test data, as shown by the ...
1
vote
1answer
16 views

Do RL agents learn the optimal “degree” of an action to take?

I have a game environment I want to train an RL model on. This environment has 2 fundamental actions that the agent can take; "Left" or "Right" (say, 0 or 1). However, the actions "Left" or "Right" ...
1
vote
1answer
31 views

Does a larger action space take longer to train an RL agent?

I am playing around with the openai gym to try and better understand reinforcement learning. One agent parameter you can modify is the action space i.e. the specific actions an agent can take in an ...
0
votes
0answers
23 views

How LSTM can be used to predict action and maximize sales

Hi would like to use LSTM with my dataset. most of people are using LSTM on NLP problem. In my case dataste look like this : IdCustomer | salesMonth_1 | action_1 |salesMonth_2 | action_2 |...
1
vote
1answer
32 views

DQN - how is it possible to train separate outputs for each action?

I'm trying to implement a Deep Q Network, but I'm stuck on how you train a network to predict multiple action-values when you can only collect data on a single action. In the paper it recommends ...
0
votes
0answers
12 views

What's the relationship amoung Temporal Difference and Policy Gradient, Deep Q learning?

I've gone through some comparisons between MC and TD to estimate $V_{\pi_\theta}(s)$. However, it seems to be not only a method to estimate V, but also a mechanism that can improve the update ...
1
vote
1answer
13 views

Why can't Policy Gradient Algorithm be seen as an Actor-Critic Method?

During the equation deducing in policy gradient algorithm(e.g., REINFORCE), we are actually using an expectancy of total reward, which we try to maximize. $$\overline{R_\theta}=E_{\tau\sim\pi_\theta}[...
1
vote
1answer
40 views

DQN - target values vs action values?

I'm trying to understand the difference between target-values and action-values in Deep Q Networks. From what I understand, action-value tries to approximate the reward of a given action (at some ...
3
votes
1answer
121 views

Why could my DDQN get significantly worse after beating the game repeatedly?

I've been trying to train a DDQN to play OpenAI Gym's CartPole-v1, but found that although it starts off well and starts getting full score (500) repeatedly (at around 600 episodes in the pic below), ...
0
votes
0answers
95 views

ValueError: Error when checking input: expected input_1 to have 4 dimensions, but got array with shape (1, 4, 4, 200, 200)

I am working on a dqn agent with a CNN which takes input of 4 images, each grey-scaled image array is of size 80x80. my model structure is like this:- ...
0
votes
0answers
12 views

Using reinforce algorithm with per-action reward instead of per-trajectory reward

I've found some articles that talk about the reinforce algorithm / monte carlo method. The algorithm boils down to using this equation. The right summation over the trajectory is the reward for the ...
0
votes
0answers
22 views

Which algorithm to use for path to prescription?

I am working on a business problem in the commercial pharma industry. In the pharma industry, we have medical representatives (Reps) selling drugs to health care providers (HCPs). They frequently ...
1
vote
1answer
42 views

What is the best way Reinforcement learning, RNN or others to predict the best action we have to take to maximize sales?

I have a dataset composed of few features : customerId, actionDay1, SalesDay1, actionDay20, SalesDay20, actionDay30, SalesDay30 action can be : call email face ...
2
votes
0answers
22 views

Which one of these is the most efficient way to model training data for a neural network that will play a snake-like game?

I am building an AI using a neural network that will play Tron against a human player. The game consists of a board with fixed width and height where each player can move at any direction (except for ...
2
votes
1answer
26 views

Average reward reinforcement learning

What is the bellman equation update rule for the average reward reinforcement learning? I searched a few articles, but could not find any practical answer.
0
votes
0answers
41 views

2019 - Bleeding edge Reinforcement Learning techniques?

I've built an RL agent using the following: Full Rainbow: Double Q-Learning (allow target network to rate the Q-score of the action selected by online network, use this score as a TD target) ...
0
votes
0answers
6 views

An inverse probability weighting reward estimator in a contextual bandit setting

I wrote a very very simple code to compare the value of the reward estimator of a new policy based on inverse propensity score, or inverse probability weighting, with the real reward of the new policy ...
2
votes
0answers
20 views

Policy Gradient methods not converging to useful mean values

I am getting familiar with Policy Gradient methods, specifically Advantage Actor Critic (A2C). My target problem use clipped continuous state and action spaces and I have therefore been training my ...
2
votes
1answer
121 views

Temporal difference learning with a neural network

Suppose I would like to train a value network $v$ via TD(0). So my TD target for a time step $t$ equals: $$R_{t+1} + \gamma v(s_{t+1})$$ If I understand correctly, I just need to use mean squared ...
0
votes
1answer
17 views

Reinforcement Learning control with known dynamic equation

I know there is model-based reinforcement learning. But all the approaches assume an MDP. If I want to do a feedback control of a system (i. e. control an inverted pendulum) it's quite easy to find ...