47

The key part of the quoted text is: To perform experience replay we store the agent's experiences $e_t = (s_t,a_t,r_t,s_{t+1})$ This means instead of running Q-learning on state/action pairs as they occur during simulation or actual experience, the system stores the data discovered for [state, action, reward, next_state] - typically in a large table. ...


43

$V^\pi(s)$ is the state-value function of MDP (Markov Decision Process). It's the expected return starting from state $s$ following policy $\pi$. In the expression $$V^\pi(s) = E_{\pi} \{G_t \vert s_t = s\} $$ $G_t$ is the total DISCOUNTED reward from time step $t$, as opposed to $R_t$ which is an immediate return. Here you are taking the expectation of ALL ...


29

Q-values are a great way to the make actions explicit so you can deal with problems where the transition function is not available (model-free). However, when your action-space is large, things are not so nice and Q-values are not so convenient. Think of a huge number of actions or even continuous action-spaces. From a sampling perspective, the ...


29

Bootstrapping in RL can be read as "using one or more estimated values in the update step for the same kind of estimated value". In most TD update rules, you will see something like this SARSA(0) update: $$Q(s,a) \leftarrow Q(s,a) + \alpha(R_{t+1} + \gamma Q(s',a') - Q(s,a))$$ The value $R_{t+1} + \gamma Q(s',a')$ is an estimate for the true value of $Q(...


17

In general we prefer to normalize the returns for stability purposes. If you work out the backpropagation equations you will see that the return affects the gradients. Thus, we would like to keep its values in a specific convenient range. We don't follow this practice for theoretical guarantees but for practical reasons. The same goes with clipping $Q$ value ...


17

Yes, but in general it is not a good tool for the task, unless there is significant feedback between predictions and ongoing behaviour of the system. To construct a reinforcement learning (RL) problem where it is worth using an RL prediction or control algorithm, then you need to identify some components: An environment that be in one of many states that ...


14

DQN suffers intrinsically from instability. In the original implementation, multiple techniques are employed to improve stability: a target network is used with parameters that lag behind the trained model; rewards are clipped to the range [-1, 1]; gradients are clipped to the range [-1, 1] (using something like Huber Loss or gradient clipping); and most ...


13

You have it right, the $V$ function gives you the value of a state, and $Q$ gives you the value of an action in a state (following a given policy $\pi$). I found the clearest explanation of Q-learning and how it works in Tom Mitchell's book "Machine Learning" (1997), ch. 13, which is downloadable. $V$ is defined as the sum of an infinite series but its not ...


12

I'd suggest you to try a hybrid approach: First, train your car in supervised fashion by demonstration. Just control it and use your commands as labels. This will let you get all the pros of SL. Then, fine tune your neural net using reinforcement learning. You don't need extra sensors for that: the rewards may be obtained from distance sensors (larger ...


11

In brief each net has a different purpose as you mentioned: The value network was used at the leaf nodes to reduce the depth of the tree search. The policy network was used to reduce the breadth of the search from a node (guiding towards promising immediate actions). In general, you can use value function methods to find an optimal policy or directly ...


11

Here you have some good references on Reinforcement Learning: Classic Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, Mass: A Bradford Book; 1998. 322 p. The draft for the second edition is available for free: Reinforcement Learning: An Introduction Russell/Norvig Chapter 21: Russell SJ, Norvig P, Davis E. Artificial intelligence: a ...


11

In my understanding, $V(s)$ is always larger than $Q(s,a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$ This is incorrect. There is not really such a thing as "the reward for current state" in the general case of a MDP. If you mean the $V(S_t)$ should include the value of $R_t$, then this is still wrong, given David ...


11

Active learning is a technique that is applied to Supervised Learning settings. In the supervised learning paradigm, you train a system by providing inputs and expected outputs (labels). The system learns to mimic the training data, ideally generalizing it to unseen but extrapolable cases. Active learning is applied normally in cases where obtaining labels ...


10

I'm no expert but it looks like AlphaGo Zero answers your question. https://deepmind.com/blog/alphago-zero-learning-scratch/ Previous versions of AlphaGo initially trained on thousands of human amateur and professional games to learn how to play Go. AlphaGo Zero skips this step and learns to play simply by playing games against itself, starting from ...


10

1. Why are the q-values of different actions very close to each other for a given state ? I'm going to explain this with a small example. Consider the game of "Catch". Fruits(circular) keep falling from the top of the screen (vertically) and the agent(square) needs to just align itself to the fruit to get the reward. There are three actions that it can ...


10

I found very intuitive the explanation of the GAE in the Supplementary material of this paper: DeepMimic. You do not need to read the paper. Just go straight to the Supplementary material section on page 143:15. For the λ-return you can find lots of information in the Reinforcement Learning book of Sutton and Barto. Hope it helps!


9

Here is my concise thought process in understanding the two different networks. First of all, the goal is to find an optimal solution (or very near-optimal) without using an exhaustive search, which is definitely challenging. Per position or state, there will be N moves possible, and on each move there will be its own depth D in a full search tree. It is ...


9

The same question has been asked to the author of the AlphaGo paper and his answer was that we don't know what would happen if AlphaGo would learn from scratch (they haven't tested it). However, given the complexity of the game, it would be a difficult task to train an algorithm from scratch without prior knowledge. Thus, it is reasonable at the beginning ...


9

Ok, it's simple! Just the "Target network approach": Select an item from Memory Bank Using Target Network, from $S_{t+1}$ determine the index of the best action $A_{t+1}$ and its Q-value Do corrections as usual The "double DQN approach"": Select an item from Memory Bank Using Online Network, from $S_{t+1}$ determine the index of the ...


8

You do not need additional learning algorithms to perform reinforcement learning in simple systems where you can explore all states. For those, simple iterative Q-learning can do very well - as well as a variety of similar techniques, such as Temporal Difference, SARSA. All these can be used without neural networks, provided your problem is not too big (...


8

As far as I understood the algorithm of AlphaGo, it is based on a simple reinforcement learning (RL) framework, using Monte-Carlo tree search to select the best actions. On the top of it, the states and actions covered by the RL algorithm are not simply the entire possible configuration of the game (Go has a huge complexity) but are based on a policy network ...


8

There are multiple ways to approach solving game playing problems. Some games can be solved by search algorithms for example. This works well for card and board games up to some level of complexity. For instance, IBM's Deep Blue was essentially a fast heuristic-driven search for optimal moves. However, probably the most generic machine learning algorithm ...


8

But in policy iteration also we are have to output a softmax vector related to each actions This is not strictly true. A softmax vector is one possible way to represent a policy, and works for discrete action spaces. The difference between policy gradient and value function approaches here is in how you use the output. For a value function you would find ...


7

There is a free online course on Reinforcement Learning by Udacity. Check : Machine Learning: Reinforcement Learning


7

It highly depends on the type of game and the information about the state of the game that is available to your AI. Some of the most famous game playing AIs from last few years are based on deep reinforcement learning (e.g. Playing Atari with Deep Reinforcement Learning), which is normal reinforcement learning (e.g. Q-learning) with a deep neural network as ...


7

The algorithm (or at least a version of it, as implemented in the Coursera RL capstone project) is as follows: Create a Replay "Buffer" that stores the last #buffer_size S.A.R.S. (State, Action, Reward, New State) experiences. Run your agent, and let it accumulate experiences in the replay-buffer until it (the buffer) has at least #batch_size experiences. ...


7

A web search for "policy collapse" "reinforcement learning" finds this question, a related one in stats.stackexchange.com and the comments section where you found the phrase. There are two other results on unrelated subjects where the words happen to appear next to each other. Then that's it - 5 results total from Google. A google books ngrams search for ...


7

I expect someone somewhere has used a RF estimator inside RL to approximate action values, if only to assess it as a comparison to other function approximators. However, it does look like from a web search that this is not used widely, and I could not find an example either. The main problem with a RL/RF hybrid with RF as the value estimator is that the ...


7

Is the "expected reward" actually $\mathcal{R}^a_{ss'}$ instead of $V^\pi(s)$? In short, yes. Although there is some context associated - $\mathcal{R}^a_{ss'}$ is in the context of specific action and state transition. You will also find $\mathcal{R}^a_{s}$ used for expected reward given only current state and action (which works fine, but moves around ...


7

Reinforcement learning is more about interacting with an environment, and while this could be posed as an RL problem, I think using Global Optimization would be a more direct approach. Essentially you want to design a cost function that describes how good a particular seating is and then use it to search the space of possible seatings. For example to solve ...


Only top voted, non community-wiki answers of a minimum length are eligible