# Questions tagged [q-learning]

A model-free reinforcement learning technique.

122 questions
Filter by
Sorted by
Tagged with
23 views

### Why DQN but no Deep Sarsa?

Why is DQN frequently used while there is hardly any occurrence of Deep Sarsa? I found this paper https://arxiv.org/pdf/1702.03118.pdf using it, but nothing else which might be relevant. I assume the ...
33 views

### How to Form the Training Examples for Deep Q Network in Reinforcement Learning?

Trying to pick up basics of reinforcement learning by self-study from some blogs and texts. Forgive me if the question is too basic and different bits that I understand are a bit messy, but even after ...
• 283
8 views

### Q-learning episode and relation to convergence in MY scenario?

I used Q-learning for routing. I have used the Bellman equation. I have certain other technical aspects in the code that add some novelty. But I have mixed doubts regarding episode and corresponding ...
1 vote
21 views

### How to create transition probability (state) for q-learning algorithm designed to control traffic light system using python?

I am trying to create a q learning algorithm to control traffic light systems. I am representing the state with a matrix : ...
48 views

### Deep Q-learning

I am working on the DDQN algorithm which is given in the following paper. I am facing a problem with the Q value. The author calculate Q value by this Q(s, a; θ , α, β) = V(s; θ , β) + A(s, a; θ , α). ...
46 views

### Q value is estimated under state V value and action A value for DDQN

How Q value is estimated under state V value and action A value. Given the below DDQN algorithm, the deep network is divided into two parts on the end layer, including state value function V(s) which ...
47 views

### Understanding DQN Algorithm

Im studying the deep q learning algorithm. You can see it in the picture here: DQN I have a few questions about the deep q learning algorithm. What do they mean with row 14: ...
32 views

### Am I using this neural network in a wrong way?

I'm trying to solve a RL problem; the Contextual Bandit problem using Deep Q Learning. My data is all simulated. I have this environment: ...
• 99
1 vote
28 views

### How to construct Q-table for complex, large and dynamic spaces in python?

I am trying to construct a Q-table. I have state space and action space. State space consists of large number of complex and dynamic number of elements, but discrete. Theoretically, I understood ...
• 111
68 views

### Efficient way to tackle card games with many q-table states?

I'm currently in the process of developing an AI for a popular card game here in Germany (called "Schafkopf"). Obviously, one could try to find a perfect strategy with the help of some game ...
22 views

### snake reinfocement learning does not learn

I’m new to reinf learning, and I’m trying this code, but the snake always goes right without learning, I can’t find the bug, could you help me? THE SNAKE CLASS ...
1 vote
79 views

### In DQN, why not use target network to predict current state Q values?

In DQN, why not use target network to predict current state Q values, and not only next state q values? In doing a basic dq learning algorithm with nn from scratch, with replay memory, and minibatch ...
84 views

### Zero Padding in Convolutional neural network

We use Convolutional neural network because it by design learns features that generalize over spatial location , so when using conv operation it reduce image size and that what we hope to have so we ...
161 views

### Exploration in Q learning: Epsilon greedy vs Exploration function

I am trying to understand how to make sure that our agent explores the state space enough before exploiting what it knows. I am aware that we use epsilon-greedy approach with a decaying epsilon to ...
1 vote
80 views

### DQN CartPole-v1 neural network doesn't optimize

I'm doing my first dnq algorithm, I'm trying to build a dnq agent, and neural network from scratch, but it seems that neural network doesn't optimize, I did 2 hidden layers, with ReLU, and the output ...
27 views

### "Learning" algorithm to use when future depends on past events (MDP property not met)

There are around 5 different retirement plans available in my country. People can pick from them freely. I would like to create a solution that would try to predict the best plan(s) given a particular ...
• 121
1 vote
38 views

### On what principle did Google's DeepMind learn to walk?

I just saw this video on Youtube. On what principle did Google's DeepMind learn to walk? Was it Q-Learning or a Genetic Algorithm or Policy Gradient?
• 23
70 views

### Dimensionality of the target for DQN agent training

From what I understand, a DQN agent has as many outputs as there are actions (for each state). If we consider a scalar state with 4 actions, that would mean that the DQN would have a 4 dimensional ...
• 21
1 vote
130 views

### Deep Q-learning in non-episodic tasks

I want to use Deep Q-learning (specifically DDQL by Hasselt et al. 2015, but it is the same principle) in a non-episodic task (continuing). I know that it is possible to use Q-learning in continuing ...
1 vote
135 views

### DQN DDQN and 3DQN differences?

I'm doing a course on reinforcement learning, and one of our tasks is to implement an agent on the Lunar lander continuous V2 environment from openAI gym. In order to solve the continuous problem, I ...
1 vote
10 views

### Implementing DQN - Pylessons.com article - Queries [closed]

I have followed the below link to implement DQN Algorithm https://pylessons.com/CartPole-reinforcement-learning/ Can someone explain me: Why do we need to have else condition in act function during ...
• 21
168 views

### Does convergence equal learning in Deep Q-learning?

In my current research project I'm using the Deep Q-learning algorithm. The setup is as follows: I'm training the model (using Deep Q-learning) on a static dataset made up of experiences extracted ...
• 81
69 views

### How to report results of RL with high variance?

I run Q-learning and SARSA algortihms on the same problem but the results fluctuate heavily and when I draw them, there is no smooth graph. How should I repost the results? I run algorithms for 500 ...
• 181
96 views

### Offline/Batch Reinforcement Learning: Doubly Robust Off-policy Estimator takes huge values

Context: My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, etc.). It is too costly for us to emulate agents. We ...
• 21
1 vote
140 views

### Definition of the Q* function in reinforcement learning

I'm making my way through Sutton's Introduction to Reinforcement Learning. He gives the definition of the $q_*$ function as follows $$q_*(a) = \mathbf{E}[R_t | A_t = a]$$ where $A_t$ is the action ...
26 views

### Reference implementation of q-learning in Python

I'm a machine learning newbie, trying to learn Q-learning. I read a few texts and I get the general gist, but what I'd really love to see is a simple example of a Q-learning algorithm in Python that I ...
• 161
1 vote
105 views

### Different Initial Q-Values in Q-Learning

When working with Q-Learning, what is the difference between having a Q_0(a) with all values zero, random or optimistic?
• 11
86 views

### When should I use normal Q learning over a DQN?

From this article here, it says that using a tabular Q function is less scalable than a deep Q network. I assume that this means that the Q table approach works for some environments, but once they ...
118 views

### Deep Q-learning, how to set q-value of non-selected actions?

I am learning Deep Q-learning by applying it to a real world problem. I have been through some tutorials and papers available online but I counldn't figure out the solution for the following problem ...
• 101
1 vote
18 views

### Wich activation function for DQL

After many research, I still can't find a neat answer about this question: When I found the loss of my state-action pair. I'm only backpropagating that loss true the network and setting all other ...
206 views

### Is this a valid stability concern/improvement for DQN/DDQN reinforcement training?

As you all know, DQN or DDQN are known for "unstable training". Let's use the well known "CartPole". The agent has to balance the stick and gets a reward of +1 per frame. You can reach the 195 ...
• 13
385 views

### How to save and load a Q-Learning Agent

I know this may sound nooby, but how do I save a Deep Q-Learning agent's progress? I mean when I close at i.e. episode 500 when my agent is trained and I restart (in my case a pygame) my agent is ...
1 vote
108 views

### Reward(t) vs. Reward(t+1) ? Reinforcement Learning, Q-learning

In "Reinforcement Learning, An Introduction; Richard S. Sutton and Andrew G. Barto" is written on page 70: 3.1 The Agent–Environment Interface At each time step t, the agent receives some ...
1 vote
101 views

### Q-learning, state transition, immediate rewards (trading logic)

I've been thinking about how to correctly calculate rewards for several weeks now. Here is a grid example: ...
40 views

• 11
1 vote
1k views

### DQN - target values vs action values?

I'm trying to understand the difference between target-values and action-values in Deep Q Networks. From what I understand, action-value tries to approximate the reward of a given action (at some ...
• 123