Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Action selection in actor-critic algorithm:

I have an action space that is just a list of values given by acts = [i for i in range(10, 100, 10)]. According to pytorch documentary, the loss is calculated as below. Could someone explain to me how ...
2
votes
1answer
36 views

Representation of state space, action space and reward system for RL porblem

I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without ...
1
vote
1answer
14 views

The meaning of γt−t0 in Reinforcement learning with pytorch

When reading pytorch tutorial: Our aim will be to train a policy that tries to maximize the discounted, cumulative reward Rt0=∑∞t=t0γt−t0rt, where ...
1
vote
0answers
14 views

Definition of obstacles in new OpenAI gym environment

I study a Reinforcement Learning algorithm that navigate an agent from one initial point to another in a complex environment where other agents and obstacles exists too. I want to make my own gym ...
1
vote
0answers
43 views

Sudden drop of score in the last few episodes

I was following this tutorial about lunar lander and deep Q learning with Tensorflow 2 and I noticed something odd. The problem was actually solved at episode 476 but then the score went from 259.90 ...
0
votes
0answers
6 views

Purpose of trace-decay parameter in eligibility traces

In TD/SARSA-lambda, eligibility traces are decayed after each step by multiplying by the discount rate and the trace-decay parameter. I understand that: The discount rate is used to reduce the value ...
1
vote
0answers
34 views

Reward(t) vs. Reward(t+1) ? Reinforcement Learning, Q-learning

In "Reinforcement Learning, An Introduction; Richard S. Sutton and Andrew G. Barto" is written on page 70: 3.1 The Agent–Environment Interface At each time step t, the agent receives some ...
0
votes
0answers
28 views

Policy Gradient with Baseline Reward Oscillation (MATLAB Reinforcement Learning Toolbox)

I'm trying to train a Policy Gradient Agent with Baseline for my RL research. I'm using the in-built RL toolbox from MATLAB (https://www.mathworks.com/help/reinforcement-learning/ug/pg-agents.html) ...
0
votes
0answers
14 views

Index tensor must have same dimensions as input tensor

I am trying to train a DQN to do optimal energy scheduling. Each state comes as a vector of 4 variables (represented by floats) saved in the replay memory as a state tensor, each action is an integer ...
1
vote
0answers
17 views

What's the definition of retraining?

In transfer learning, we always use new data to retrain the pre-trained model. But, what is the specific and official definition of retraining? Or what papers mentioned this definition, in transfer ...
0
votes
0answers
48 views

Q-learning, state transition, immediate rewards (trading logic)

I've been thinking about how to correctly calculate rewards for several days now. Here is a grid example: ...
0
votes
1answer
27 views

When should the last action be included in the state in reinforcement learning?

I am having some confusion as to whether the action should be included as part of the state input to an agent in a reinforcement learning setting (state-action pair). As from my observation, this is ...
1
vote
0answers
18 views

Deep Reinforcement Learning [closed]

I am new in this field and I am trying to understand the Deep Deterministic Policy Gradient (DDPG) model. I can not understand the use of the target network in this model. Why do we need to define the ...
0
votes
0answers
15 views

find the parameter of model with Q learning

I have a question with regard to Q learning. I am a beginner in Q learning. Every example that I saw is related to the environment that the goal is assigned to a place. (like cliff walk that we know ...
0
votes
0answers
27 views

Reinforcement algorithm for Trading

I am trying to implement a Reinforcement Learning Algorithm in a trading scenario. It seems natural to me to use final profit as the reward. However, in this scenario, for each trading episode, there ...
0
votes
0answers
17 views

Reinforcement Learning : Why acting greedily with the optimal value function gives you the optimal policy?

The course of David Silver about Reinforcement Learning explains how you get the optimal policy from the optimal value function. It seems to be very simple, you just have to act greedily, by ...
6
votes
1answer
47 views

Reinforcement Learning: Policy Gradient derivation question

I have been reading this excellent post: https://medium.com/@jonathan_hui/rl-policy-gradients-explained-9b13b688b146 and following the RL-videos by David Silver, and I did not get this thing: For $\...
4
votes
1answer
104 views

Hindsight Experience Replay (HER) results obtained 50 times faster than original paper?

I am reproducing the results from Hindsight Experience Replay by Andrychowicz et. al. In the original paper they present the results below, where the agent is trained for 200 epochs. 200 epochs * 800 ...
0
votes
0answers
16 views

Multi Agent Reinforcement learning with communincation between the agents

I am trying to build reinforcement learning agents using DQN and policy gradient algorithms and OpenAI. There needs to some exchange of information that needs to happen between these agents to share ...
0
votes
0answers
8 views

Binary value as observation in Observation space Stable Baselines

How would a binary value (0,1) make sense in an observation space that is between 0 and 1? In Stable baselines I use a continuous observation space between 0 and 1. ...
2
votes
0answers
15 views

Markov Decision Process representation

I'm attempting to model a simple process using a Markov Decision Process. Let $A$ be a set of $3$ actions : $ A \in \{b,s\}$. $T(s,a,s')$ represents the probability of if in state $s$ , take action $...
-2
votes
2answers
44 views

Simulating mental disorders in machine learning systems

Is there any work as the following example in ML: Suppose a reinforcement learning system which has a fixed penalty for every ...
0
votes
1answer
13 views

How is the target_f updated in the Keras solution to the Deep Q-learning Cartpole/Gym algorithm?

There's a popular solution to the CartPole game using Keras and Deep Q-Learning: https://keon.github.io/deep-q-learning/ But there's a line of code that's confusing, this same question has been asked ...
4
votes
2answers
35 views

Q learning for blackjack, reward function?

I am currently learning reinforcement learning and am have built a blackjack game. There is an obvious reward at the end of the game (payout), however some actions do not directly lead to rewards (...
1
vote
1answer
31 views

Dynamic pricing with aggregate constraints

so I have this situation. I am trying to understand whether my customers will buy my product at a certain price based on a previous offer made to them. Specifically, I have a lot of data on my clients ...
1
vote
0answers
82 views

Deep reinforcement learning with multi-dimensional action

I am trying to design reinforcement learning algorithm. My action and state space are continuous. Action, which I would like to take can be represented by a matrix, lets say of dimension $n \times n$. ...
2
votes
0answers
22 views

Actions taken by agentn/ agent performance not improving

Hi I am trying to develop an rl agent using PPO algorithm. My agent takes an action(CFM) to maintain a state variable called RAT in between 24 to 24.5. I am using PPO algorithm of stable-baselines ...
1
vote
1answer
42 views

Assistance needed on what machine learning approach to use

👋 I'm currently writing my Master's Thesis on Subjective tagging of sounds and I feel that I've been stuck with the same problem for quite a time now and need assistance to progress. I'll, in short, ...
1
vote
0answers
21 views

Cold-start problem in Real Time Bidding

I'm currently on the reading stage of the deployment of an RTB system. I've seen the problem of a cold start (having no initial guess of how to bid) in several papers, but I haven't really seen it be ...
3
votes
1answer
38 views

Why replay memory store old states and action rather than Q-value (Deep Q-learning)

Here is the algorithm use in Google's DeepMind Atari paper The replay memory D store transition (old_state, action performed, reward, new_state) The old_state and the performed action a are needed ...
1
vote
1answer
25 views

Idenitity between TD(0) algorithm and Policy Evaluation in Dynamic Programming when alpha is equal to 1

TD(0) algorithm is defined as the iterative update of the following: $$ V(s) \leftarrow V(s) + \alpha({r + \gamma V(s')} - V(s) ) $$ Now, if we assume alpha to be equal to 1, we get the traditional ...
2
votes
2answers
109 views

Deep Q Network gives same Q values and doesn't improve

I'm trying to build a deep Q network to play snake. I've run into an issue where the agent doesn't learn and its performance at the end of the training cycle is to repeatedly kill itself. After a bit ...
4
votes
1answer
27 views

Reinforcement learning with sparse acting agent

I'm working on a problem where the optimal policy involves the agent "doing nothing" most of the time, and "doing something" during rare critical moments. Is there any literature or best practices ...
2
votes
1answer
72 views

Would Deep Q Learning work for a finite horizon problem?

I want to apply Deep Q Learning to a problem, which has a clear finite horizon definition, like: $$V(s) = \mathbb{E}[r_1 + r_2]$$ Since the horizon is finite, I do not use reward discounting. My ...
0
votes
0answers
25 views

Reinforcement Learning techniques to use if there is a direct connecction between (state, action) and reward

I want to build a model. The input state is a combination of different numbers in sequence (Their orders matter to the final result). The effectiveness of such number combination can be computed into ...
1
vote
0answers
24 views

How can I improve this tensorflow 2.0 function?

I recreated some code I found online for solving the bandits problem using policy gradient. The example was in tensorflow 1.0 so I recreated it with tensorflow 2.0 using eager execution and gradient ...
1
vote
1answer
36 views

Multiply Tensorflow sequential layer by fixed integers

I'm trying to make a simple reinforcement learning model that makes one of three decisions, A, B, or ...
0
votes
0answers
13 views

Vowpal Wabbit, Daemon mode “write error: Bad file descriptor”

I'm using the Vowpal Image provided by AWS @ amazonaws.com/sagemaker-rl-vw-container:vw-8.7.0-cpu. However, when I run Vowpal in daemon mode ...
0
votes
0answers
48 views

Why is orthogonal weights initialization so important for PPO?

I have implemented PPO to solve Atari environments. For the longest time I couldn't figure out why my model would not converge as fast as other open source solutions. Eventually it boiled down to this ...
2
votes
0answers
26 views

Incentivizing curiosity in a sparse reward environment

I'm quite new to reinforcement learning, but have been exploring different kinds of architectures (DQN, dueling DQN, actor critic, etc.) and evaluating their ability to solve certain problems. The ...
1
vote
0answers
16 views

Reinforcement learning converges for mean loss but not for each training data

Here I show a dummy example that represents my actual problem. My neural network (NN) receives one input and gives the probabilities for two output nodes. The code for the NN is: ...
2
votes
1answer
21 views

reinforcement learning: Decompose a policy gradient

I am studying the policy gradient through the website: https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f Couldn't figure out how the first equation becomes the second ...
2
votes
1answer
38 views

Trouble understanding the partial differentiation used in reinforcement learning

I am studying deterministic actor-critic algorithms in reinforcement learning. I try to give a brief explanation of actor-critic algorithms before jumping into the mathematics. The actor takes in ...
6
votes
2answers
96 views

Actor-critic architecture: How is the policy updated?

I am going through the ddpg baseline code to try and gain an intuitive understanding of how the actor and critic networks function. DDPG has two components: the actor which is the deterministic ...
0
votes
0answers
17 views

Assumptions on discounted long-term loss

The infinite horizon discounted long-term loss is defined as: $$ f(\theta) = \mathbb{E}_{\tau \sim \mathbb{P}(.|\theta)}\left[\sum_{t=1}^{\infty}{\gamma^t l_m(s_t,a_t)}\right]$$ where $(s_t,a_t) \in ...
0
votes
0answers
16 views

exploitation vs. exploration: upper-confidence-bound vs. epsilon-greedy

I am looking into some different ways for doing exploitation vs. exploration (e.g. multi-arm bandit problem). There are approaches like upper_confidence_bound and epsilon-greedy. I am wondering what ...
0
votes
0answers
189 views

reinforcement learning: PPO vs. DDPG vs. TRPO - difference and intuition

I know there is a lot of blog talk about the PPO, DDPG and TRPO, but I am wondering would it be possible to explain the differences of these methods in layman's term? What's the intuition behind them ...
1
vote
0answers
15 views

Rewards are converged but with a lot of variations

I am training a reinforcement learning agent on an episodic task of fixed episode length. I am tracking the training process by plotting the cumulative rewards over an episode. I am using tensorboard ...
1
vote
0answers
15 views

Deep Q Learning - training slows down significantly

I'm trying to build a deep Q network to play snake. I designed the game so that the window is 600 by 600 and the snake's head moves 30 pixels each tick. I implemented the DQN algorithm with memory ...
1
vote
0answers
9 views

Autonomous learning - chatbots

My chatbots need to be trained when we get new data or feedbacks from users. Can someone provides ways how these chatbots can learn on themselves and become intelligent day by day? Some of techniques ...

1
2 3 4 5
9