Questions tagged [reinforcement-learning]
Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.
619
questions
1
vote
0
answers
120
views
Q-learning, state transition, immediate rewards (trading logic)
I've been thinking about how to correctly calculate rewards for several weeks now.
Here is a grid example:
...
- 21
2
votes
1
answer
1k
views
When should the last action be included in the state in reinforcement learning?
I am having some confusion as to whether the action should be included as part of the state input to an agent in a reinforcement learning setting (state-action pair). As from my observation, this is ...
- 21
1
vote
0
answers
22
views
Deep Reinforcement Learning [closed]
I am new in this field and I am trying to understand the Deep Deterministic Policy Gradient (DDPG) model. I can not understand the use of the target network in this model. Why do we need to define the ...
- 111
0
votes
2
answers
236
views
Reinforcement algorithm for Trading [closed]
I am trying to implement a Reinforcement Learning Algorithm in a trading scenario.
It seems natural to me to use final profit as the reward. However, in this scenario, for each trading episode, there ...
- 1
1
vote
2
answers
113
views
Reinforcement Learning : Why acting greedily with the optimal value function gives you the optimal policy?
The course of David Silver about Reinforcement Learning explains how you get the optimal policy from the optimal value function.
It seems to be very simple, you just have to act greedily, by ...
- 11
6
votes
1
answer
138
views
Reinforcement Learning: Policy Gradient derivation question
I have been reading this excellent post: https://medium.com/@jonathan_hui/rl-policy-gradients-explained-9b13b688b146 and following the RL-videos by David Silver, and I did not get this thing:
For $\...
- 63
4
votes
1
answer
211
views
Hindsight Experience Replay (HER) results obtained 50 times faster than original paper?
I am reproducing the results from Hindsight Experience Replay by Andrychowicz et. al. In the original paper they present the results below, where the agent is trained for 200 epochs.
200 epochs * 800 ...
- 311
2
votes
1
answer
85
views
Markov Decision Process representation
I'm attempting to model a simple process using a Markov Decision Process.
Let $A$ be a set of $3$ actions : $ A \in \{b,s\}$.
$T(s,a,s')$ represents the probability of if in state $s$ , take action $...
- 207
-2
votes
2
answers
61
views
Simulating mental disorders in machine learning systems
Is there any work as the following example in ML:
Suppose a reinforcement learning system which has a fixed penalty for every ...
- 107
2
votes
1
answer
109
views
How is the target_f updated in the Keras solution to the Deep Q-learning Cartpole/Gym algorithm?
There's a popular solution to the CartPole game using Keras and Deep Q-Learning:
https://keon.github.io/deep-q-learning/
But there's a line of code that's confusing, this same question has been asked ...
- 23
4
votes
2
answers
283
views
Q learning for blackjack, reward function?
I am currently learning reinforcement learning and am have built a blackjack game.
There is an obvious reward at the end of the game (payout), however some actions do not directly lead to rewards (...
- 181
1
vote
1
answer
152
views
Dynamic pricing with aggregate constraints
so I have this situation. I am trying to understand whether my customers will buy my product at a certain price based on a previous offer made to them. Specifically, I have a lot of data on my clients ...
- 159
1
vote
0
answers
283
views
Deep reinforcement learning with multi-dimensional action
I am trying to design reinforcement learning algorithm. My action and state space are continuous. Action, which I would like to take can be represented by a matrix, lets say of dimension $n \times n$. ...
2
votes
0
answers
52
views
Actions taken by agentn/ agent performance not improving
Hi I am trying to develop an rl agent using PPO algorithm. My agent takes an action(CFM) to maintain a state variable called RAT in between 24 to 24.5. I am using PPO algorithm of stable-baselines ...
- 555
1
vote
1
answer
64
views
Assistance needed on what machine learning approach to use
👋
I'm currently writing my Master's Thesis on Subjective tagging of sounds and I feel that I've been stuck with the same problem for quite a time now and need assistance to progress. I'll, in short, ...
- 113
1
vote
0
answers
88
views
Cold-start problem in Real Time Bidding
I'm currently on the reading stage of the deployment of an RTB system. I've seen the problem of a cold start (having no initial guess of how to bid) in several papers, but I haven't really seen it be ...
- 111
3
votes
1
answer
286
views
Why replay memory store old states and action rather than Q-value (Deep Q-learning)
Here is the algorithm use in Google's DeepMind Atari paper
The replay memory D store transition (old_state, action performed, reward, new_state)
The old_state and the performed action a are needed ...
- 352
2
votes
2
answers
116
views
Idenitity between TD(0) algorithm and Policy Evaluation in Dynamic Programming when alpha is equal to 1
TD(0) algorithm is defined as the iterative update of the following:
$$ V(s) \leftarrow V(s) + \alpha({r + \gamma V(s')} - V(s) ) $$
Now, if we assume alpha to be equal to 1, we get the traditional ...
2
votes
2
answers
215
views
Deep Q Network gives same Q values and doesn't improve
I'm trying to build a deep Q network to play snake. I've run into an issue where the agent doesn't learn and its performance at the end of the training cycle is to repeatedly kill itself. After a bit ...
- 79
5
votes
1
answer
208
views
Reinforcement learning with sparse acting agent
I'm working on a problem where the optimal policy involves the agent "doing nothing" most of the time, and "doing something" during rare critical moments. Is there any literature or best practices ...
- 71
2
votes
1
answer
797
views
Would Deep Q Learning work for a finite horizon problem?
I want to apply Deep Q Learning to a problem, which has a clear finite horizon definition, like:
$$V(s) = \mathbb{E}[r_1 + r_2]$$
Since the horizon is finite, I do not use reward discounting. My ...
- 203
0
votes
0
answers
25
views
Reinforcement Learning techniques to use if there is a direct connecction between (state, action) and reward
I want to build a model. The input state is a combination of different numbers in sequence (Their orders matter to the final result). The effectiveness of such number combination can be computed into ...
1
vote
1
answer
119
views
How to improve tensorflow 2.0 code for policy gradient?
I recreated some code I found online for solving the bandits problem using policy gradient. The example was in tensorflow 1.0 so I recreated it with tensorflow 2.0 using eager execution and gradient ...
- 111
1
vote
1
answer
781
views
Multiply Tensorflow sequential layer by fixed integers
I'm trying to make a simple reinforcement learning model that makes one of three decisions, A, B, or ...
- 181
5
votes
1
answer
2k
views
Why is orthogonal weights initialization so important for PPO?
I have implemented PPO to solve Atari environments. For the longest time I couldn't figure out why my model would not converge as fast as other open source solutions. Eventually it boiled down to this ...
- 797
2
votes
0
answers
58
views
Incentivizing curiosity in a sparse reward environment
I'm quite new to reinforcement learning, but have been exploring different kinds of architectures (DQN, dueling DQN, actor critic, etc.) and evaluating their ability to solve certain problems. The ...
- 71
3
votes
1
answer
303
views
Maximum Entropy Policy Gradient Derivation
I am reading through the paper on Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review by Sergey Levine. I am having a difficulty in understanding this part of the ...
3
votes
1
answer
158
views
reinforcement learning: Decompose a policy gradient
I am studying the policy gradient through the website: https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f
Couldn't figure out how the first equation becomes the second ...
- 2,755
2
votes
1
answer
263
views
Trouble understanding the partial differentiation used in reinforcement learning
I am studying deterministic actor-critic algorithms in reinforcement learning.
I try to give a brief explanation of actor-critic algorithms before jumping into the mathematics. The actor takes in ...
- 311
6
votes
2
answers
1k
views
Actor-critic architecture: How is the policy updated?
I am going through the ddpg baseline code to try and gain an intuitive understanding of how the actor and critic networks function.
DDPG has two components: the actor which is the deterministic ...
- 311
1
vote
0
answers
802
views
reinforcement learning: PPO vs. DDPG vs. TRPO - difference and intuition
I know there is a lot of blog talk about the PPO, DDPG and TRPO, but I am wondering would it be possible to explain the differences of these methods in layman's term? What's the intuition behind them ...
- 2,755
1
vote
0
answers
83
views
Rewards are converged but with a lot of variations
I am training a reinforcement learning agent on an episodic task of fixed episode length. I am tracking the training process by plotting the cumulative rewards over an episode. I am using tensorboard ...
- 555
1
vote
0
answers
36
views
Deep Q Learning - training slows down significantly
I'm trying to build a deep Q network to play snake. I designed the game so that the window is 600 by 600 and the snake's head moves 30 pixels each tick. I implemented the DQN algorithm with memory ...
- 79
2
votes
0
answers
23
views
Autonomous learning - chatbots
My chatbots need to be trained when we get new data or feedbacks from users. Can someone provides ways how these chatbots can learn on themselves and become intelligent day by day?
Some of techniques ...
- 894
0
votes
2
answers
173
views
Reinforcement Learning in NLP for chatbots
Is anyone aware of any successful implementation of reinforcement learning for NLP. I am looking to for chatbots which can learn automatically.
Tried searching internet but found very few articles ...
- 894
2
votes
1
answer
541
views
Having a reward structure which gives high positive rewards compared to the negative rewards
I am training an RL agent using PPO algorithm for a control problem. The objective of the agent is to maintain temperature in a room. It is an episodic task with episode length of 9 hrs and step size(...
- 555
0
votes
0
answers
46
views
DNN loss gets smaller but accuracy stays the same
I am learning a DeepNN to choose between three decisions in a simulation. Therefore, I can run the simulation as often as I want and can generate as many samples as I want. Based on this tutorial (...
- 51
1
vote
1
answer
70
views
Formulation of a reward structure
I am new to reinforcement learning and experimenting with training of RL agents.
I have a doubt about reward formulation, from a given state if a agent takes a good action i give a positive reward, ...
- 555
1
vote
0
answers
308
views
Actor Critic Model implementation
I am going to work on a project which requires implementation of A2C model using Tensorflow 2.0. I am new in the Machine Learning field and also in Python. These are topics which I have covered ...
- 113
1
vote
1
answer
3k
views
Training a model that has both 2D and 1D features using a CNN
I'm looking to pre-train a model for an RL agent but I'm having some trouble figuring some stuff out.
Dataset: Minerl MineRLNavigateDense-v0
The observation space includes :
2D screen input (64,64) ...
- 111
0
votes
1
answer
194
views
How to handle differences between training and deploying of an RL agent
Hi I am training an RL agent for a control problem. The objective of the agent is to maintain temperature in a zone. It is an episodic task with episode length of 10 hrs and actions being taken every ...
- 555
1
vote
0
answers
45
views
How can I increase the speed and performance of my implementation of an AI for Reversi?
I made an AI for Reversi, aka Othello (8×8), like Alpha Zero, using this book. This book is written in Japanese. The source code of the AI I implemented can be found in this Github repository. There ...
- 11
1
vote
0
answers
463
views
DQN vs PG - when to use which?
I'd like to understand when using PG methods is more adequate than using DQN methods. Just to give a bit of background: I am currently using both APEX and R2D2 for my projects. Both work very well in ...
- 797
1
vote
1
answer
1k
views
Different results every time I train a reinforcement learning agent
I am training an RL agent for a control problem using PPO algorithm. I am using stable-baselines library for it.
The objective of an agent is to maintain a temperature of 24 deg in a zone and it ...
- 555
3
votes
1
answer
5k
views
Evaluating a trained Reinforcement Learning Agent?
I am new to reinforcement learning agent training. I have read about PPO algorithm and used stable baselines library to train an agent using PPO. So my question here is how do I evaluate a trained RL ...
- 555
1
vote
0
answers
186
views
Transitioning from Math PhD to ML research [closed]
I am currently a Math PhD about to defend in January. I work in a field in functional analysis that uses a lot of measure theory (but no stats).
I have been considering transitioning careers since I ...
- 139
1
vote
2
answers
411
views
Deep Q network disregards input, giving identical output no matter the input state
I've creating a very simple game,
The board is an array of size 6.
0 is "empty cell" , 5 is "Goal", 8 is "player location"
[8 0 0 5 0 0] for example means the agent needs to move 2 "right" to win.
...
- 151
2
votes
2
answers
3k
views
What are regret bounds?
I searched for the term and it appears in a few articles but it is used without explanation. The only explanation I could find is in a PhD thesis: "Regret bounds are the common thread in the analysis ...
- 21
2
votes
1
answer
312
views
transform a supervised neural network to reinforcement learning?
I have a functional LSTM model that works with an acceptable performance. How can I now convert this supervised model to a reinforcement learning model for improving the performane? Is there any ...
- 21
2
votes
1
answer
6k
views
Policy Gradient with continuous action space
How to apply reinforce/policy-gradient algorithms for continuous action space. I have learnt that one of the advantages of policy gradients is , it is applicable for continuous action space. One way I ...
- 555