Questions tagged [q-learning]
A model-free reinforcement learning technique.
0
votes
0answers
14 views
Openai Spaces for a modified environment
I have a 2-dimensional array of normalized data. I am using
space = np.array([0,1,...366],[0,0.000001,.....1])
I need to fit this as an observation space in ...
0
votes
0answers
5 views
Q learning transition matrix trouble
First, apologies if this isn't the right forum for my question -- let me know if there's a better place.
I'm trying to implement Q learning on a two-dimensional grid world, where the agent has four ...
0
votes
0answers
20 views
Q-Learning where some actions are more difficult to predict than others
I'm trying to train a deep Q network to optimize play in a game. For simplicity, let's say my game only has two actions, A and B.
The reward distribution for A is somewhat uniform in the range -1 to ...
0
votes
1answer
24 views
Why is “next state” kept in RL experience replay?
Following this explanation on what is experience replay (and others), I noticed an experience element is defined as
$e_t = (s_t,a_t,r_t,s_{t+1})$
My question is, why do we need the ...
0
votes
0answers
7 views
Is every value-iteration off-policy DP is a Q-learning?
Is it true to say that every value-iteration off-policy DP is a Q-learning technique. or there are many more specific definition for it?
1
vote
0answers
20 views
What is the meaning of the Variant Q-learning and To what INPUT and OUTPUT refer? in Abstract of DeepMind DQN paper 2013
-INPUT and OUTPUT OF ATARI DQN:
In the abstract paragraph of the DQN work by DeepMind https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf it has written:
" We present the first deep learning model to ...
2
votes
1answer
99 views
What is the difference between dynamic programming and Q-learning?
What is the difference between the DP-based algorithm and Q-learning?
0
votes
1answer
22 views
How to represent an image as state in a Q-table
I'm trying to do Q-learning with the Atari games using the gym python's package.
I want to use the image as the state of my algorithm, but I came up with a doubt: ...
-1
votes
0answers
32 views
How to interpret/understand Q-table in FrozenLake
I am again to invite attention to the example code for FrozenLake at this question link. The actual code is from here. The full code is again reproduced below. I am unable to interpret the final Q-...
-1
votes
0answers
24 views
Q update rule of Q-learning and DQN
As far as I know,
tabular Q-learning updates its Q-value as follows:
\begin{equation}
Q(s,a) \leftarrow Q(s,a) + \alpha (\gamma + \max_{a'}Q(s',a') - Q(s,a))
\end{equation}
where $\gamma + \max_{...
0
votes
1answer
22 views
How does Q-Learning deal with mixed strategies?
I'm trying to understand how Q-learning deals with games where the optimal policy is a mixed strategy. The Bellman equation says that you should choose $max_a(Q(s,a))$ but this implies a single unique ...
0
votes
1answer
39 views
Why not use max(returns) instead of average(returns) in off-policy Monte Carlo control?
As I understood it, in reinforcement learning off-policy Monte Carlo control, when estimating the state-action value function Q(s, a), it is estimated as an weighted average of observed returns.
...
-1
votes
0answers
9 views
Enforcing a Q learning neural network function with the inclusion of Q values in input space
I have been experimenting with deep Q learning on a trading task (any task, if that matters): my Medium article
I use double dueling noisy layer architecture that looks as follows:
I got an idea of ...
1
vote
1answer
60 views
Reinforcement learning: negative reward (punish) illegal actions?
If you train an agent using reinforcement learning (with Q-function in this case), should you give a negative reward (punish) if the agent proposes illegal actions for the presented state?
I guess ...
0
votes
1answer
332 views
IndexError: index 804 is out of bounds for axis 0 with size 800
i installed a self driving car project from superdatascience site , when i open the map using terminal after a while the map window close up or it closes directly after i maximize the map window and ...
1
vote
1answer
38 views
Will reinforcement learning work if states wont get repeated again?
I am working on a information retrieval model where the user enters a query and the model has to retrieve 3 most relevant FAQ pairs.I am collecting implicit feedback in terms of page clicks etc.What I ...
1
vote
0answers
46 views
Dueling DQN - Calculation of Q-value
I'm trying to implement a Double Dueling DQN on LunarLander and I'm facing an issue as my model is not learning so I'm trying to debug the graph and this leads me to a question regarding the ...
1
vote
1answer
47 views
What's going wrong with my Tic Tac Toe Q-Learning Alghoritm?
my Q-learning alghoritm currently choices the "sub optimal" option and not the best one.
...
1
vote
1answer
67 views
What is the immediate reward in value iteration?
Suppose you're given an MDP where rewards are attributed for reaching a state, independently of the action. Then when doing value iteration:
$$ V_{i+1} = \max_a \sum_{s'} P_a(s,s') (R_a(s,s') + \...
3
votes
1answer
557 views
Reinforcement learning: decreasing loss without increasing reward
I'm trying to solve OpenAI Gym's LunarLander-v2.
I'm using the Deep Q-Learning algorithm. I have tried various hyperparameters, but I can't get a good score.
Generally the loss decreases over many ...
2
votes
1answer
39 views
How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )
How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )?
I want to replace the DNN section of Qlearning with a RF or SVR but the problem is that there is no clear ...
1
vote
0answers
64 views
Deep Q-Learning with large number of actions
I'm using DQN with large number of actions in [0, 10000, step = 1000]. This means I have an action space of size 11 (including 0 and 10000). Action space is still discrete. My problem is that, instead ...
5
votes
1answer
78 views
Representing similar states in reinforcement learning?
Let's say I'd like to design a Q learning algorithm that learns to play poker. The number of different possible States is very large, but a lot are very similar: for example, if the initial state ...
2
votes
1answer
80 views
Calculate Q parameter for Deep Q-Learning applied to videogames
I am working on Deep Q-learning applied to Snake, and I am confused on the methodology. Based on the DeepMind paper on the topic and other sources, the Q-value with the Bellman equation needs to be ...
1
vote
1answer
43 views
Tflearn “nan” weight matrices
I wanted to build a DQN. So I followed this code and watched some videos about the idea of DQN. My Code is this (mine is written in tflearn and his in keras):
...
2
votes
1answer
119 views
Dueling DQN what does a' mean?
what does $a'$ mean in the "combining" equation in Dueling DQN? (top of the page 5)
$$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(...
3
votes
1answer
409 views
Prioritized Experience Replay - why to approximate the Density Function?
I am reading about Prioritized Experience Replay, and can't understand the following:
On page 4, every transition can be selected from the table with its own probability.
Here is the cumulative ...
1
vote
1answer
32 views
Adding a bias makes Q-learning algorithm ineffective
I've been working through the Q-Network learning example in this Arthur Juliani's blog. It's based on the pretty trivia Open Gym Frozen Lake example. It's base implementation get's about 47% success ...
3
votes
1answer
35 views
Why does Q-learning use an actor model and critic model?
I'm currently reading Hands on Machine Learning with Scikit-Learn & Tensorflow, and I'm wondering why does Q-learning require an actor model and a critic model to learn?
On page 465, it states:
...
1
vote
1answer
54 views
Experience Replay, must return minibatch back to Memory Bank?
During Experience Replay, we are randomly gathering a minibatch from the Memory bank. We then use the minibatch to correct our NeuralNetwork q-value function approximator.
When done, should we return ...
1
vote
0answers
109 views
Experience Replay Explain
I have read many blog articles, research papers and watched many youtube videos, but it seems it is hard to find why experience replay is efficient.
I know that the experience replay stores (state, ...
0
votes
1answer
123 views
Can you interpolate with QLearning or Reinforcement learning in general?
I am currently researching the usages of machine learning paradigms for pathfinding problems. I am currently looking into the reinforcement learning paradigm and I used QLearning for pathfinding.
...
3
votes
1answer
53 views
What is the optimal value of a Markov Decision process with Single actions at each state?
I am trying to solve some questions about a MRP (i.e. a Markov Decision process with only one possible action at each state). The setup is as follows:
There are two states ($a$ and $b$) stepping to $...
1
vote
1answer
37 views
Policy gradient on data only, without emulators
It is too costly for my team to emulate the agent (executing the action and assessing the reward), meaning our only option is to learn the optimal policy on our dataset. The good thing is that we have ...
1
vote
1answer
91 views
Reinforcement Learning with static state
Can Q Learning work with a static state for each step?
What I mean by that is that the actions do not influence the following state at all. The episodes just iterate over the same data over and over ...
1
vote
1answer
42 views
How does a Q algorithm consider future rewards?
I am trying to understand the underlying logic of Q learning (deep Q learning to be precise). At the moment I am stuck at the notion of future rewards.
To understand the logic, I am reviewing some of ...
7
votes
2answers
500 views
Is this a Q-learning algorithm or just brute force?
I have been playing with an algorithm that learns how to play tictactoe. The basic pseudocode is:
...
2
votes
1answer
254 views
Q learning neural network experience replay problem
I am currently trying to create a tic tac toe q learning neural network to introduce me to reinforcement learning, however it didn't work so I decided to try a simpler project requiring a network to ...
2
votes
1answer
1k views
Reinforcement Learning on data only (NO emulators)
My team and I started digging into RL for the purpose of a specific application. We have plenty of data of an agent carrying out suboptimal policies (states and rewards...).
It is too costly for us ...
1
vote
0answers
140 views
2
votes
1answer
111 views
Q-learning why do we subtract the Q(s, a) term during update?
I can't understand the meaning of $-Q(s_t, a_t)$ term in the Q-learning algorithm, and can't find explanation to it either.
Everything else makes sence. The q-learning algorithm is an off-policy ...
1
vote
1answer
335 views
Q learning Neural network Tic tac toe - When to train net
This is another question I have on a q learning neural network being used to win tic tac toe, which is that im not sure i understand when to actually back propogate through the network.
What i am ...
3
votes
1answer
115 views
Neural network q learning for tic tac toe - how to use the threshold
I am currently programming a q learning neural network tha does not work. I have previously asked a question about inputs and have sorted that out. My current idea to why the program does not work is ...
3
votes
0answers
109 views
Graphical results of Q-Learning: is improvement possible by parameter tweaking?
From left to right:
Maximum Q value for action selection (averaged)
Train error (averaged)
Reward from environment (averaged)
I run double Q-learning.
A behavioral policy is ε-greedy, ε constant ...
1
vote
1answer
446 views
Q Learning Neural network for tic tac toe Input implementation problem
I've recently become interested in machine learning, specifically neural networks, and after creating ones to solve basic problems such as XOR and Sin and Cos graphs, however i am now looking into ...
2
votes
1answer
2k views
Is my understanding of On-Policy and Off-Policy TD algorithms correct?
After reading several questions here and browsing some pages on the topic, here is my understanding of the key difference between Q-learning (as an example of off-policy) and SARSA (as an example of ...
1
vote
1answer
1k views
Negative Rewards and Activation Functions
I have a question regarding appropriate activation functions with environments that have both positive and negative rewards.
In reinforcement learning, our output, I believe, should be the expected ...
1
vote
0answers
40 views
Isn't the optimizer network in deepminds learning to learn a DRQN?
In the paper "Learning to learn by gradient descent by gradient descent" they describe an RNN which learns gradient transformation to learn an optimizer.
The optimizer network directly interacts ...
3
votes
1answer
812 views
Why random sample from replay for DQN?
I'm trying to gain an intuitive understanding of deep reinforcement learning. In deep Q-networks (DQN) we store all actions/environments/rewards in a memory array and at the end of the episode, "...
3
votes
1answer
103 views
Clamping Q function to it's theoretical maximum, yes or no?
I'm implementing DQN algorithm from scratch on MountainCar simulation.
I'm using a setup of $reward = 1.0$ when car hits the flag, and $0$ otherwise.
Reward decay factor is set to $\gamma=0.99$. ...
