Questions tagged [q-learning]

A model-free reinforcement learning technique.

Filter by
Sorted by
Tagged with
1
vote
1answer
406 views

Reinforcement learning: negative reward (punish) illegal actions?

If you train an agent using reinforcement learning (with Q-function in this case), should you give a negative reward (punish) if the agent proposes illegal actions for the presented state? I guess ...
2
votes
1answer
2k views

IndexError: index 804 is out of bounds for axis 0 with size 800

i installed a self driving car project from superdatascience site , when i open the map using terminal after a while the map window close up or it closes directly after i maximize the map window and ...
1
vote
1answer
60 views

Will reinforcement learning work if states wont get repeated again?

I am working on a information retrieval model where the user enters a query and the model has to retrieve 3 most relevant FAQ pairs.I am collecting implicit feedback in terms of page clicks etc.What I ...
1
vote
0answers
107 views

Dueling DQN - Calculation of Q-value

I'm trying to implement a Double Dueling DQN on LunarLander and I'm facing an issue as my model is not learning so I'm trying to debug the graph and this leads me to a question regarding the ...
2
votes
1answer
109 views

What's going wrong with my Tic Tac Toe Q-Learning Alghoritm?

my Q-learning alghoritm currently choices the "sub optimal" option and not the best one. ...
1
vote
1answer
325 views

What is the immediate reward in value iteration?

Suppose you're given an MDP where rewards are attributed for reaching a state, independently of the action. Then when doing value iteration: $$ V_{i+1} = \max_a \sum_{s'} P_a(s,s') (R_a(s,s') + \...
7
votes
1answer
4k views

Reinforcement learning: decreasing loss without increasing reward

I'm trying to solve OpenAI Gym's LunarLander-v2. I'm using the Deep Q-Learning algorithm. I have tried various hyperparameters, but I can't get a good score. Generally the loss decreases over many ...
5
votes
1answer
4k views

RL Advantage function why A = Q-V instead of A=V-Q?

In RL Course by David Silver - Lecture 7: Policy Gradient Methods, David explains what an Advantage function is, and how it's the difference between Q(s,a) and the V(s) Preliminary, from this post: ...
2
votes
1answer
69 views

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )

How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )? I want to replace the DNN section of Qlearning with a RF or SVR but the problem is that there is no clear ...
1
vote
0answers
361 views

Deep Q-Learning with large number of actions

I'm using DQN with large number of actions in [0, 10000, step = 1000]. This means I have an action space of size 11 (including 0 and 10000). Action space is still discrete. My problem is that, instead ...
7
votes
2answers
146 views

Representing similar states in reinforcement learning?

Let's say I'd like to design a Q learning algorithm that learns to play poker. The number of different possible States is very large, but a lot are very similar: for example, if the initial state ...
2
votes
1answer
119 views

Calculate Q parameter for Deep Q-Learning applied to videogames

I am working on Deep Q-learning applied to Snake, and I am confused on the methodology. Based on the DeepMind paper on the topic and other sources, the Q-value with the Bellman equation needs to be ...
1
vote
1answer
85 views

Tflearn “nan” weight matrices

I wanted to build a DQN. So I followed this code and watched some videos about the idea of DQN. My Code is this (mine is written in tflearn and his in keras): ...
2
votes
1answer
221 views

Dueling DQN what does a' mean?

what does $a'$ mean in the "combining" equation in Dueling DQN? (top of the page 5) $$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(...
4
votes
2answers
718 views

Prioritized Experience Replay - why to approximate the Density Function?

I am reading about Prioritized Experience Replay, and can't understand the following: On page 4, every transition can be selected from the table with its own probability. Here is the cumulative ...
1
vote
1answer
42 views

Adding a bias makes Q-learning algorithm ineffective

I've been working through the Q-Network learning example in this Arthur Juliani's blog. It's based on the pretty trivia Open Gym Frozen Lake example. It's base implementation get's about 47% success ...
3
votes
1answer
50 views

Why does Q-learning use an actor model and critic model?

I'm currently reading Hands on Machine Learning with Scikit-Learn & Tensorflow, and I'm wondering why does Q-learning require an actor model and a critic model to learn? On page 465, it states: ...
2
votes
1answer
81 views

Experience Replay, must return minibatch back to Memory Bank?

During Experience Replay, we are randomly gathering a minibatch from the Memory bank. We then use the minibatch to correct our NeuralNetwork q-value function approximator. When done, should we return ...
1
vote
0answers
177 views

Experience Replay Explain

I have read many blog articles, research papers and watched many youtube videos, but it seems it is hard to find why experience replay is efficient. I know that the experience replay stores (state, ...
0
votes
1answer
203 views

Can you interpolate with QLearning or Reinforcement learning in general?

I am currently researching the usages of machine learning paradigms for pathfinding problems. I am currently looking into the reinforcement learning paradigm and I used QLearning for pathfinding. ...
3
votes
1answer
68 views

What is the optimal value of a Markov Decision process with Single actions at each state?

I am trying to solve some questions about a MRP (i.e. a Markov Decision process with only one possible action at each state). The setup is as follows: There are two states ($a$ and $b$) stepping to $...
1
vote
1answer
50 views

Policy gradient on data only, without emulators

It is too costly for my team to emulate the agent (executing the action and assessing the reward), meaning our only option is to learn the optimal policy on our dataset. The good thing is that we have ...
1
vote
1answer
198 views

Reinforcement Learning with static state

Can Q Learning work with a static state for each step? What I mean by that is that the actions do not influence the following state at all. The episodes just iterate over the same data over and over ...
1
vote
1answer
63 views

How does a Q algorithm consider future rewards?

I am trying to understand the underlying logic of Q learning (deep Q learning to be precise). At the moment I am stuck at the notion of future rewards. To understand the logic, I am reviewing some of ...
8
votes
2answers
1k views

Is this a Q-learning algorithm or just brute force?

I have been playing with an algorithm that learns how to play tictactoe. The basic pseudocode is: ...
5
votes
1answer
376 views

Q learning - how to use experience replay, when playing against other agent?

I am currently trying to create a tic tac toe q learning neural network to introduce me to reinforcement learning, however it didn't work so I decided to try a simpler project requiring a network to ...
3
votes
1answer
2k views

Reinforcement Learning on data only (NO emulators)

My team and I started digging into RL for the purpose of a specific application. We have plenty of data of an agent carrying out suboptimal policies (states and rewards...). It is too costly for us ...
1
vote
0answers
214 views

Simple Q-learning neural network using numpy

...
2
votes
1answer
306 views

Q-learning why do we subtract the Q(s, a) term during update?

I can't understand the meaning of $-Q(s_t, a_t)$ term in the Q-learning algorithm, and can't find explanation to it either. Everything else makes sence. The q-learning algorithm is an off-policy ...
1
vote
1answer
395 views

Q learning Neural network Tic tac toe - When to train net

This is another question I have on a q learning neural network being used to win tic tac toe, which is that im not sure i understand when to actually back propogate through the network. What i am ...
3
votes
1answer
142 views

Neural network q learning for tic tac toe - how to use the threshold

I am currently programming a q learning neural network tha does not work. I have previously asked a question about inputs and have sorted that out. My current idea to why the program does not work is ...
3
votes
0answers
136 views

Graphical results of Q-Learning: is improvement possible by parameter tweaking?

From left to right: Maximum Q value for action selection (averaged) Train error (averaged) Reward from environment (averaged) I run double Q-learning. A behavioral policy is ε-greedy, ε constant ...
1
vote
1answer
594 views

Q Learning Neural network for tic tac toe Input implementation problem

I've recently become interested in machine learning, specifically neural networks, and after creating ones to solve basic problems such as XOR and Sin and Cos graphs, however i am now looking into ...
3
votes
1answer
4k views

Is my understanding of On-Policy and Off-Policy TD algorithms correct?

After reading several questions here and browsing some pages on the topic, here is my understanding of the key difference between Q-learning (as an example of off-policy) and SARSA (as an example of ...
1
vote
1answer
2k views

Negative Rewards and Activation Functions

I have a question regarding appropriate activation functions with environments that have both positive and negative rewards. In reinforcement learning, our output, I believe, should be the expected ...
1
vote
0answers
44 views

Isn't the optimizer network in deepminds learning to learn a DRQN?

In the paper "Learning to learn by gradient descent by gradient descent" they describe an RNN which learns gradient transformation to learn an optimizer. The optimizer network directly interacts ...
3
votes
2answers
2k views

Why random sample from replay for DQN?

I'm trying to gain an intuitive understanding of deep reinforcement learning. In deep Q-networks (DQN) we store all actions/environments/rewards in a memory array and at the end of the episode, "...
3
votes
1answer
186 views

Clamping Q function to it's theoretical maximum, yes or no?

I'm implementing DQN algorithm from scratch on MountainCar simulation. I'm using a setup of $reward = 1.0$ when car hits the flag, and $0$ otherwise. Reward decay factor is set to $\gamma=0.99$. ...
2
votes
1answer
571 views

Q learning and Neural Network for Tic Tac Toe

I have been working on a tic-tac-toe assignment for my Robot Learning class. We were asked to program a tic-tac-toe game and assign; +1 if X wins, -1 if O wins and 0 it the game results with a draw. ...
6
votes
1answer
3k views

Simple Q-Table Learning: Understanding Example Code

I'm trying to follow a tutorial for Q-Table learning from this source, and am having difficulty understanding a small piece of the code. Here's the entire block: ...
3
votes
1answer
689 views

Neural Network Learning Rate vs Q-Learning Learning Rate

I'm just getting into machine learning--mostly Reinforcement Learning--using a neural network trained on Q-values. However, in looking at the hyper-parameters, there are two that seem redundant: the ...
9
votes
2answers
2k views

Why does Q Learning diverge?

My Q-Learning algorithm's state values keep on diverging to infinity, which means my weights are diverging too. I use a neural network for my value-mapping. I've tried: Clipping the "reward + ...
1
vote
0answers
526 views

Keras not converging to optimum while TensorFlow does

I'm working on a Reinforcement learning project where the agent needs to navigate itself around the maze and get to the goal. (I used Q Learning as my algorithm) The agent found the optimal path in 50 ...
5
votes
1answer
1k views

Keras input dimension bug?

Keras has a problem with the input dimension. My first layer looks like this: ...
0
votes
1answer
465 views

Q-learning with a state-action-state reward structure and a Q-matrix with states as rows and actions as columns

I have set up a Q-learning problem in R, and would like some help with the theoretical correctness of my approach in framing the problem. Problem structure For this problem, the environment consists ...
28
votes
2answers
37k views

What is “experience replay” and what are its benefits?

I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
0
votes
1answer
1k views

Choosing the right parameters for SARSA and Q-Learning & Comparing Models

What is the correct way to fine-tune a model's (SARSA(0), SARSA(lambda), Q(0), Q(lambda)) parameters, and how can one compare the models? I read that typically one compares the number of actions ...
4
votes
1answer
2k views

Multiple Output Layers in Neural Networks in Deep Q Learning

I am trying to train a computer to play with LEGO bricks (simulated) using DQN. My input is an image with 4 color channels (RGB and depth) and the output of the neural network is the coordinates of ...
6
votes
1answer
7k views

Understanding advantage functions

The paper explaining 'Advantage Updating' as a method to improve Q-learning uses the following as its motivation. Q-learning requires relatively little computation per update, but it is useful to ...
2
votes
0answers
887 views

Initial Q-values in Q-Learning

I am running a Q-learning algorithm with a finite time horizon. Are 'optimistic initial conditions' still preferred if there is a possibility that some states will not be visited multiple times? ...