Questions tagged [q-learning]
A model-free reinforcement learning technique.
127
questions
1
vote
0
answers
43
views
reinforcement learning reward choices
To start with, this is not a homework thing. In my attempt to finally get a practical working knowledge of table based re-inforcement learning, I came up with a very silly and easy dice game, serving ...
- 111
0
votes
0
answers
6
views
"Unique" or "Repeated" experiences in memory replay...?
I'm training an RL agent/model (DRL/DQN).
Say that, for each learning step, the memory replay used by the agent to learn, has N elements (experiences) stored, where only X are unique elements (...
0
votes
1
answer
50
views
Can I compare Q-VALUE for the same action across different states?
Dears,
I'm new with RL and try to apply to my project.
I've run RL with some example data and got the
My question is if I could compare the Q-values for the same current action across different prior ...
- 1
0
votes
1
answer
68
views
Deep Q-Learning: How are network parameters updated, and why consider episodes in the first place?
I'm trying to wrap my head around the implementation of deep $Q$-learning, and why we even consider episodes in the first place. The usual set-up is that we initialize some starting state $s_0$, then ...
- 101
1
vote
1
answer
58
views
In Q-learning, why does Q index on both state and action?
In Q-learning, Q is an array of expected rewards for (state, action) combinations.
It seems to me the same result could be achieved while slightly simplifying the algorithm, if instead of associating ...
- 159
1
vote
1
answer
408
views
In a finite horizon reinforcement learning problem, are the $Q$ and value functions dependent on time?
Typically the definition I see for the $Q$ and value functions is
$$
Q^\pi(s_t, a_t) = \mathbb{E}_\tau\left[\sum_{t'=t}^T\gamma^{t'-t}r(s_{t'}, a_{t'})\ |\ s_t, a_t\right] \\
V^\pi(s_t) = \mathbb{E}_\...
- 113
0
votes
1
answer
33
views
Bayesian state description in Reinforcement Learning
What's the best approach to feed a bayesian description of an observed state to a Reinforcement Learning agent?
Brief context: I have an agent situated in an environment, which it perceives through a ...
- 3
0
votes
1
answer
171
views
Reinforcement Learning in a game against itself?
Let's we have a tictactoe design using RL against a random player. We can describe the system by enhancing and giving rewards to good actions. But what if the Rl model is played with itself? What ...
- 1
0
votes
1
answer
195
views
Why DQN but no Deep Sarsa?
Why is DQN frequently used while there is hardly any occurrence of Deep Sarsa?
I found this paper https://arxiv.org/pdf/1702.03118.pdf using it, but nothing else which might be relevant.
I assume the ...
- 3
2
votes
1
answer
197
views
How to Form the Training Examples for Deep Q Network in Reinforcement Learning?
Trying to pick up basics of reinforcement learning by self-study from some blogs and texts. Forgive me if the question is too basic and different bits that I understand are a bit messy, but even after ...
- 335
1
vote
0
answers
66
views
How to create transition probability (state) for q-learning algorithm designed to control traffic light system using python?
I am trying to create a q learning algorithm to control traffic light systems.
I am representing the state with a matrix :
...
- 11
0
votes
1
answer
66
views
Deep Q-learning
I am working on the DDQN algorithm which is given in the following paper.
I am facing a problem with the Q value.
The author calculate Q value by this
Q(s, a; θ , α, β) = V(s; θ , β) + A(s, a; θ , α).
...
- 11
0
votes
0
answers
67
views
Q value is estimated under state V value and action A value for DDQN
How Q value is estimated under state V value and action A value.
Given the below DDQN algorithm, the deep network is divided into two parts on the end layer, including state value function V(s) which ...
- 11
0
votes
1
answer
132
views
Understanding DQN Algorithm
Im studying the deep q learning algorithm. You can see it in the picture here:
DQN
I have a few questions about the deep q learning algorithm.
What do they mean with row 14: ...
1
vote
0
answers
96
views
How to construct Q-table for complex, large and dynamic spaces in python?
I am trying to construct a Q-table.
I have state space and action space. State space consists of large number of complex and dynamic number of elements, but discrete.
Theoretically, I understood ...
- 111
2
votes
1
answer
171
views
Efficient way to tackle card games with many q-table states?
I'm currently in the process of developing an AI for a popular card game here in Germany (called "Schafkopf"). Obviously, one could try to find a perfect strategy with the help of some game ...
- 63
1
vote
1
answer
197
views
In DQN, why not use target network to predict current state Q values?
In DQN, why not use target network to predict current state Q values, and not only next state q values?
In doing a basic dq learning algorithm with nn from scratch, with replay memory, and minibatch ...
0
votes
2
answers
260
views
Zero Padding in Convolutional neural network
We use Convolutional neural network because it by design learns features that generalize over spatial location , so when using conv operation it reduce image size and that what we hope to have so we ...
4
votes
1
answer
565
views
Exploration in Q learning: Epsilon greedy vs Exploration function
I am trying to understand how to make sure that our agent explores the state space enough before exploiting what it knows. I am aware that we use epsilon-greedy approach with a decaying epsilon to ...
- 55
1
vote
0
answers
212
views
DQN CartPole-v1 neural network doesn't optimize
I'm doing my first dnq algorithm, I'm trying to build a dnq agent, and neural network from scratch, but it seems that neural network doesn't optimize, I did 2 hidden layers, with ReLU, and the output ...
2
votes
0
answers
31
views
"Learning" algorithm to use when future depends on past events (MDP property not met)
There are around 5 different retirement plans available in my country. People can pick from them freely. I would like to create a solution that would try to predict the best plan(s) given a particular ...
- 121
3
votes
1
answer
123
views
On what principle did Google's DeepMind learn to walk?
I just saw this video on Youtube.
On what principle did Google's DeepMind learn to walk?
Was it Q-Learning or a Genetic Algorithm or Policy Gradient?
- 43
2
votes
1
answer
162
views
Dimensionality of the target for DQN agent training
From what I understand, a DQN agent has as many outputs as there are actions (for each state). If we consider a scalar state with 4 actions, that would mean that the DQN would have a 4 dimensional ...
- 21
1
vote
0
answers
270
views
Deep Q-learning in non-episodic tasks
I want to use Deep Q-learning (specifically DDQL by Hasselt et al. 2015, but it is the same principle) in a non-episodic task (continuing). I know that it is possible to use Q-learning in continuing ...
- 11
1
vote
0
answers
280
views
DQN DDQN and 3DQN differences?
I'm doing a course on reinforcement learning, and one of our tasks is to implement an agent on the Lunar lander continuous V2 environment from openAI gym.
In order to solve the continuous problem, I ...
- 11
1
vote
0
answers
11
views
Implementing DQN - Pylessons.com article - Queries [closed]
I have followed the below link to implement DQN Algorithm
https://pylessons.com/CartPole-reinforcement-learning/
Can someone explain me:
Why do we need to have else condition in act function during ...
- 21
2
votes
1
answer
458
views
Does convergence equal learning in Deep Q-learning?
In my current research project I'm using the Deep Q-learning algorithm. The setup is as follows: I'm training the model (using Deep Q-learning) on a static dataset made up of experiences extracted ...
- 81
0
votes
1
answer
353
views
How to report results of RL with high variance?
I run Q-learning and SARSA algortihms on the same problem but the results fluctuate heavily and when I draw them, there is no smooth graph. How should I repost the results? I run algorithms for 500 ...
- 201
2
votes
0
answers
127
views
Offline/Batch Reinforcement Learning: Doubly Robust Off-policy Estimator takes huge values
Context:
My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, etc.).
It is too costly for us to emulate agents. We ...
- 21
1
vote
1
answer
381
views
Definition of the Q* function in reinforcement learning
I'm making my way through Sutton's Introduction to Reinforcement Learning. He gives the definition of the $q_*$ function as follows
$$
q_*(a) = \mathbf{E}[R_t | A_t = a]
$$
where $A_t$ is the action ...
- 13
2
votes
1
answer
749
views
Reference implementation of q-learning in Python
I'm a machine learning newbie, trying to learn Q-learning. I read a few texts and I get the general gist, but what I'd really love to see is a simple example of a Q-learning algorithm in Python that I ...
- 255
1
vote
1
answer
485
views
Different Initial Q-Values in Q-Learning
When working with Q-Learning, what is the difference between having a Q_0(a) with all values zero, random or optimistic?
- 11
0
votes
1
answer
327
views
When should I use normal Q learning over a DQN?
From this article here, it says that using a tabular Q function is less scalable than a deep Q network. I assume that this means that the Q table approach works for some environments, but once they ...
1
vote
1
answer
267
views
Deep Q-learning, how to set q-value of non-selected actions?
I am learning Deep Q-learning by applying it to a real world problem. I have been through some tutorials and papers available online but I counldn't figure out the solution for the following problem ...
- 111
1
vote
0
answers
28
views
Wich activation function for DQL
After many research, I still can't find a neat answer about this question:
When I found the loss of my state-action pair. I'm only backpropagating that loss true the network and setting all other ...
0
votes
1
answer
319
views
Is this a valid stability concern/improvement for DQN/DDQN reinforcement training?
As you all know, DQN or DDQN are known for "unstable training".
Let's use the well known "CartPole". The agent has to balance the stick and gets a reward of +1 per frame. You can reach the 195 ...
- 13
1
vote
1
answer
2k
views
How to save and load a Q-Learning Agent
I know this may sound nooby, but how do I save a Deep Q-Learning agent's progress? I mean when I close at i.e. episode 500 when my agent is trained and I restart (in my case a pygame) my agent is ...
- 13
1
vote
0
answers
153
views
Reward(t) vs. Reward(t+1) ? Reinforcement Learning, Q-learning
In "Reinforcement Learning, An Introduction; Richard S. Sutton and Andrew G. Barto" is written on page 70:
3.1 The Agent–Environment Interface
At each time step t, the agent receives some ...
- 21
1
vote
0
answers
124
views
Q-learning, state transition, immediate rewards (trading logic)
I've been thinking about how to correctly calculate rewards for several weeks now.
Here is a grid example:
...
- 21
2
votes
1
answer
91
views
Markov Decision Process representation
I'm attempting to model a simple process using a Markov Decision Process.
Let $A$ be a set of $3$ actions : $ A \in \{b,s\}$.
$T(s,a,s')$ represents the probability of if in state $s$ , take action $...
- 207
1
vote
0
answers
43
views
help understanding deep Q learning algorithm from deep mind paper
I'm studying this paper https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf, and I have a question about proposed algorithm(it appears on page 7 of the paper):
The problem I see is ...
- 11
2
votes
1
answer
116
views
How is the target_f updated in the Keras solution to the Deep Q-learning Cartpole/Gym algorithm?
There's a popular solution to the CartPole game using Keras and Deep Q-Learning:
https://keon.github.io/deep-q-learning/
But there's a line of code that's confusing, this same question has been asked ...
- 23
2
votes
2
answers
238
views
Deep Q Network gives same Q values and doesn't improve
I'm trying to build a deep Q network to play snake. I've run into an issue where the agent doesn't learn and its performance at the end of the training cycle is to repeatedly kill itself. After a bit ...
- 79
2
votes
1
answer
846
views
Would Deep Q Learning work for a finite horizon problem?
I want to apply Deep Q Learning to a problem, which has a clear finite horizon definition, like:
$$V(s) = \mathbb{E}[r_1 + r_2]$$
Since the horizon is finite, I do not use reward discounting. My ...
- 203
2
votes
0
answers
59
views
Incentivizing curiosity in a sparse reward environment
I'm quite new to reinforcement learning, but have been exploring different kinds of architectures (DQN, dueling DQN, actor critic, etc.) and evaluating their ability to solve certain problems. The ...
- 71
1
vote
0
answers
36
views
Deep Q Learning - training slows down significantly
I'm trying to build a deep Q network to play snake. I designed the game so that the window is 600 by 600 and the snake's head moves 30 pixels each tick. I implemented the DQN algorithm with memory ...
- 79
1
vote
0
answers
554
views
Reducing the training time of an RL agent
I am trying to develop an rl agent using DQN algorithm.During training, the agent interacts with environment which is a simulated one.Each episode takes around 10 mins to run. This way if want my ...
- 555
1
vote
1
answer
234
views
How to formulate reward of an rl agent with two objectives
I have started learning reinforcement learning and trying to apply it for my use case. I am developing an rl agent which can maintain temperature at a particular value, and minimize the energy ...
- 555
1
vote
0
answers
71
views
Keras high loss and high accuracy in gk bot with reinforcement learning?
I'm making goal-keeper bot in haxball game. It worked well when i trained less but i worked worse when i trained more.
Last reinforcement state: 5160 episode - 4171281 steps - 0.05 epsilon:
Last fit ...
1
vote
1
answer
1k
views
Q-learning when minimising a total cost instead of maximising a total reward
I have a decision problem where the results are measured as a cost that I want to minimise. It seems like a good fit to Q-learning, but I am not sure how to adjust it to deal with a cost instead of a ...
- 75