Questions tagged [q-learning]
A model-free reinforcement learning technique.
0
votes
0answers
1 view
When immigrate from DP to MDP and when we immigrate from a MDP (Q-learning) to DQN?
In the following of my previous question : Difference between DP and Q-learning?
When we immigrate from DP to MDP and when we immigrate from a MDP (Q-learning) to DQN?
I have some problem in the ...
0
votes
0answers
4 views
input and output of DQN
I have two related questions first one is the inputs and outputs of my problem? whereas the second one is related to this concept written in the abstract paragraph of first work in DQN (2013 DeepMind)...
2
votes
1answer
28 views
Difference between DP and Q-learning?
What is the difference between DP based algorithm and Q-learning one?
0
votes
1answer
18 views
How to represent an image as state in a Q-table
I'm trying to do Q-learning with the Atari games using the gym python's package.
I want to use the image as the state of my algorithm, but I came up with a doubt: ...
0
votes
0answers
16 views
How to interpret/understand Q-table in FrozenLake
I am again to invite attention to the example code for FrozenLake at this question link. The actual code is from here. The full code is again reproduced below. I am unable to interpret the final Q-...
0
votes
0answers
13 views
Q update rule of Q-learning and DQN
As far as I know,
tabular Q-learning updates its Q-value as follows:
\begin{equation}
Q(s,a) \leftarrow Q(s,a) + \alpha (\gamma + \max_{a'}Q(s',a') - Q(s,a))
\end{equation}
where $\gamma + \max_{...
0
votes
1answer
13 views
How does Q-Learning deal with mixed strategies?
I'm trying to understand how Q-learning deals with games where the optimal policy is a mixed strategy. The Bellman equation says that you should choose $max_a(Q(s,a))$ but this implies a single unique ...
0
votes
1answer
20 views
Why not use max(returns) instead of average(returns) in off-policy Monte Carlo control?
As I understood it, in reinforcement learning off-policy Monte Carlo control, when estimating the state-action value function Q(s, a), it is estimated as an weighted average of observed returns.
...
0
votes
0answers
5 views
Enforcing a Q learning neural network function with the inclusion of Q values in input space
I have been experimenting with deep Q learning on a trading task (any task, if that matters): my Medium article
I use double dueling noisy layer architecture that looks as follows:
I got an idea of ...
1
vote
1answer
44 views
Reinforcement learning: negative reward (punish) illegal actions?
If you train an agent using reinforcement learning (with Q-function in this case), should you give a negative reward (punish) if the agent proposes illegal actions for the presented state?
I guess ...
-1
votes
0answers
27 views
Q-Learning: Dynamic alpha vs fixed alpha
I guess this is a basic question, but I have seen that instead of fixing the value of $\alpha\in[0, 1]$ when updating the Q-value, some set it to $\alpha=1/k$, where $k$ is the number of times a state-...
0
votes
0answers
180 views
Reinforcement Learning for Cart Pole problem in OpenAI Gym
I have a very specific question about mapping the Observations to states in a POMDP problem. Once I get there, I can construct a Q-table with states as rows and actions as columns and use Q-Learning.
...
0
votes
1answer
140 views
IndexError: index 804 is out of bounds for axis 0 with size 800
i installed a self driving car project from superdatascience site , when i open the map using terminal after a while the map window close up or it closes directly after i maximize the map window and ...
1
vote
1answer
28 views
Will reinforcement learning work if states wont get repeated again?
I am working on a information retrieval model where the user enters a query and the model has to retrieve 3 most relevant FAQ pairs.I am collecting implicit feedback in terms of page clicks etc.What I ...
1
vote
0answers
31 views
Dueling DQN - Calculation of Q-value
I'm trying to implement a Double Dueling DQN on LunarLander and I'm facing an issue as my model is not learning so I'm trying to debug the graph and this leads me to a question regarding the ...
1
vote
1answer
43 views
What's going wrong with my Tic Tac Toe Q-Learning Alghoritm?
my Q-learning alghoritm currently choices the "sub optimal" option and not the best one.
...
1
vote
1answer
40 views
What is the immediate reward in value iteration?
Suppose you're given an MDP where rewards are attributed for reaching a state, independently of the action. Then when doing value iteration:
$$ V_{i+1} = \max_a \sum_{s'} P_a(s,s') (R_a(s,s') + \...
3
votes
1answer
279 views
Reinforcement learning: decreasing loss without increasing reward
I'm trying to solve OpenAI Gym's LunarLander-v2.
I'm using the Deep Q-Learning algorithm. I have tried various hyperparameters, but I can't get a good score.
Generally the loss decreases over many ...
2
votes
1answer
34 views
How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )
How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )?
I want to replace the DNN section of Qlearning with a RF or SVR but the problem is that there is no clear ...
1
vote
0answers
44 views
Deep Q-Learning with large number of actions
I'm using DQN with large number of actions in [0, 10000, step = 1000]. This means I have an action space of size 11 (including 0 and 10000). Action space is still discrete. My problem is that, instead ...
5
votes
1answer
72 views
Representing similar states in reinforcement learning?
Let's say I'd like to design a Q learning algorithm that learns to play poker. The number of different possible States is very large, but a lot are very similar: for example, if the initial state ...
2
votes
1answer
75 views
Calculate Q parameter for Deep Q-Learning applied to videogames
I am working on Deep Q-learning applied to Snake, and I am confused on the methodology. Based on the DeepMind paper on the topic and other sources, the Q-value with the Bellman equation needs to be ...
1
vote
1answer
34 views
Tflearn “nan” weight matrices
I wanted to build a DQN. So I followed this code and watched some videos about the idea of DQN. My Code is this (mine is written in tflearn and his in keras):
...
2
votes
1answer
114 views
Dueling DQN what does a' mean?
what does $a'$ mean in the "combining" equation in Dueling DQN? (top of the page 5)
$$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(...
3
votes
1answer
356 views
Prioritized Experience Replay - why to approximate the Density Function?
I am reading about Prioritized Experience Replay, and can't understand the following:
On page 4, every transition can be selected from the table with its own probability.
Here is the cumulative ...
1
vote
1answer
31 views
Adding a bias makes Q-learning algorithm ineffective
I've been working through the Q-Network learning example in this Arthur Juliani's blog. It's based on the pretty trivia Open Gym Frozen Lake example. It's base implementation get's about 47% success ...
3
votes
1answer
34 views
Why does Q-learning use an actor model and critic model?
I'm currently reading Hands on Machine Learning with Scikit-Learn & Tensorflow, and I'm wondering why does Q-learning require an actor model and a critic model to learn?
On page 465, it states:
...
1
vote
1answer
50 views
Experience Replay, must return minibatch back to Memory Bank?
During Experience Replay, we are randomly gathering a minibatch from the Memory bank. We then use the minibatch to correct our NeuralNetwork q-value function approximator.
When done, should we return ...
1
vote
0answers
104 views
Experience Replay Explain
I have read many blog articles, research papers and watched many youtube videos, but it seems it is hard to find why experience replay is efficient.
I know that the experience replay stores (state, ...
0
votes
1answer
99 views
Can you interpolate with QLearning or Reinforcement learning in general?
I am currently researching the usages of machine learning paradigms for pathfinding problems. I am currently looking into the reinforcement learning paradigm and I used QLearning for pathfinding.
...
3
votes
1answer
53 views
What is the optimal value of a Markov Decision process with Single actions at each state?
I am trying to solve some questions about a MRP (i.e. a Markov Decision process with only one possible action at each state). The setup is as follows:
There are two states ($a$ and $b$) stepping to $...
1
vote
1answer
30 views
Policy gradient on data only, without emulators
It is too costly for my team to emulate the agent (executing the action and assessing the reward), meaning our only option is to learn the optimal policy on our dataset. The good thing is that we have ...
0
votes
0answers
21 views
reinforcement learning-robot sticks in terminal
I wanna write a simple code of Q learning in Matlab but the robot stick in terminal I don't know how can I update Q(6,6) in Q matrix?[enter link description here][1]
...
1
vote
1answer
83 views
Reinforcement Learning with static state
Can Q Learning work with a static state for each step?
What I mean by that is that the actions do not influence the following state at all. The episodes just iterate over the same data over and over ...
1
vote
1answer
40 views
How does a Q algorithm consider future rewards?
I am trying to understand the underlying logic of Q learning (deep Q learning to be precise). At the moment I am stuck at the notion of future rewards.
To understand the logic, I am reviewing some of ...
6
votes
2answers
421 views
Is this a Q-learning algorithm or just brute force?
I have been playing with an algorithm that learns how to play tictactoe. The basic pseudocode is:
...
2
votes
1answer
228 views
Q learning neural network experience replay problem
I am currently trying to create a tic tac toe q learning neural network to introduce me to reinforcement learning, however it didn't work so I decided to try a simpler project requiring a network to ...
0
votes
0answers
6 views
May online TD learner perform better in very non-stationary environments?
Consider these environment mappings:
simple
somewhat noisy and smoothly-trendy
very noisy and sharply-trendy
For the first two my realization of an experience-replay double Q-learning agent works ...
2
votes
1answer
920 views
Reinforcement Learning on data only (NO emulators)
My team and I started digging into RL for the purpose of a specific application. We have plenty of data of an agent carrying out suboptimal policies (states and rewards...).
It is too costly for us ...
1
vote
0answers
126 views
2
votes
1answer
79 views
Q-learning why do we subtract the Q(s, a) term during update?
I can't understand the meaning of $-Q(s_t, a_t)$ term in the Q-learning algorithm, and can't find explanation to it either.
Everything else makes sence. The q-learning algorithm is an off-policy ...
0
votes
0answers
108 views
How to implement Qlearning for cartpole?
I am implementing Q learning to solve the cartpole from OpenAI gym with tensorflow. I think I have a correct implementation, but the algorithm does not seem to be learning anything. It seems I must ...
1
vote
1answer
310 views
Q learning Neural network Tic tac toe - When to train net
This is another question I have on a q learning neural network being used to win tic tac toe, which is that im not sure i understand when to actually back propogate through the network.
What i am ...
3
votes
1answer
109 views
Neural network q learning for tic tac toe - how to use the threshold
I am currently programming a q learning neural network tha does not work. I have previously asked a question about inputs and have sorted that out. My current idea to why the program does not work is ...
3
votes
0answers
104 views
Graphical results of Q-Learning: is improvement possible by parameter tweaking?
From left to right:
Maximum Q value for action selection (averaged)
Train error (averaged)
Reward from environment (averaged)
I run double Q-learning.
A behavioral policy is ε-greedy, ε constant ...
1
vote
1answer
409 views
Q Learning Neural network for tic tac toe Input implementation problem
I've recently become interested in machine learning, specifically neural networks, and after creating ones to solve basic problems such as XOR and Sin and Cos graphs, however i am now looking into ...
2
votes
1answer
2k views
Is my understanding of On-Policy and Off-Policy TD algorithms correct?
After reading several questions here and browsing some pages on the topic, here is my understanding of the key difference between Q-learning (as an example of off-policy) and SARSA (as an example of ...
1
vote
1answer
1k views
Negative Rewards and Activation Functions
I have a question regarding appropriate activation functions with environments that have both positive and negative rewards.
In reinforcement learning, our output, I believe, should be the expected ...
1
vote
0answers
40 views
Isn't the optimizer network in deepminds learning to learn a DRQN?
In the paper "Learning to learn by gradient descent by gradient descent" they describe an RNN which learns gradient transformation to learn an optimizer.
The optimizer network directly interacts ...
3
votes
1answer
680 views
Why random sample from replay for DQN?
I'm trying to gain an intuitive understanding of deep reinforcement learning. In deep Q-networks (DQN) we store all actions/environments/rewards in a memory array and at the end of the episode, "...
