10
$\begingroup$

For some reason, AlphaGo Zero isn't getting as much publicity as the original AlphaGo, despite its incredible results. Starting from scratch, it's already beaten AlphaGo Master and has passed numerous other benchmarks. Even more incredibly, it's done this in 40 days. Google names it as "arguably the best Go player in the world".

DeepMind claims this is a "novel form of reinforcement learning" - is this technique truly novel? Or have there been other times when this technique was used- and if so, what were their results? I think the requirements I'm talking about are 1) no human intervention and 2) no historical play, but these are flexible.

This appears to be a similar question, but all the answers seem to start from the assumption that AlphaGo Zero is the first of its kind.

$\endgroup$
6
  • $\begingroup$ Reinforcement learning is not new. What techniques Google claimed that they are the first one? $\endgroup$
    – SmallChess
    Oct 20, 2017 at 0:14
  • $\begingroup$ There's a quote about it on the linked website, and in the article they use the phrase "The neural network in AlphaGo Zero is trained from games of self-play by a novel reinforcement learning algorithm." $\endgroup$
    – Dubukay
    Oct 20, 2017 at 0:18
  • 1
    $\begingroup$ Self-playing is definitely not new. It existed before Google. There're details in their algorithm that make them "novel". Maybe someone else can answer. $\endgroup$
    – SmallChess
    Oct 20, 2017 at 0:21
  • 2
    $\begingroup$ I understand that- I guess I'm trying to understand what made their approach so incredibly good, and whether that's something we should expect to see in other areas. Is it a new philosophy or just really good code? $\endgroup$
    – Dubukay
    Oct 20, 2017 at 0:23
  • 1
    $\begingroup$ I found a copy of the paper here: nature.com/articles/… (includes share access token, which is from the blog that links it, so it is legit public share AFAICS). Even after reading the description though it is hard to pick out the actual novelty - all the individual ideas seem to be pre-existing RL/game-playing techniques, it may just be specific combination of them that is novel $\endgroup$ Oct 20, 2017 at 11:58

1 Answer 1

6
$\begingroup$

The AlphaGo Zero article from Nature, "Mastering the Game of Go without Human Knowledge", claims four major differences from the earlier version:

  1. Self-learning only (not trained on human games)
  2. Using only the board and stones as input (no hand-written features).
  3. Using a single neural network for policies and values
  4. A new tree-search algorithm that uses this combined policy/value network to guide where to search for good moves.

Points (1) and (2) are not new in Reinforcement learning, but improve on the previous AlphaGo software as stated in the comments to your question. It just means they are now using pure Reinforcement Learning starting from randomly initialized weights. This is enabled by better, faster learning algorithms.

Their claim here is "Our primary contribution is to demonstrate that superhuman performance can be achieved without human domain knowledge." (p. 22).

Points (3) and (4) are novel in the sense that their algorithm is simpler and more general than their previous approach. They also mention that is is an improvement on previous work by Guo et al.

Unifying the policy/value network (3) enables them to implement a more efficient variant of Monte-Carlo tree search to search for good moves and simultaneous using the search tree to train the network faster (4). This is very powerful.

Furthermore, they describe a number of interesting implementation details like batching and reusing data-structures to optimize the search for new moves.

The effect is that it needs less computing power, running on 4 TPUs rather than 176 GPUs and 48 TPUs for previous versions of their software.

This definitely makes it "novel" in the context of Go software. I believe that (3) and (4) are also "novel" in a broader context and will be applicable in other Reinforcement Learning domains such as e.g. robotics.

$\endgroup$
1
  • $\begingroup$ I think (4) is alluded to in David Silver's lectures - lecture 10 on classic games - in a lot of existing cases the MCTS is guided by the already-trained ML. In the case of AlphaGo Zero, this is flipped around and the result of the MCTS is used to set the learning targets for the ML. However, the thing that makes me wonder whether it is truly "novel" is the possibility of doing just that is mentioned in the lecture . . . $\endgroup$ Oct 22, 2017 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.