# Why doesn't AlphaGo expect its opponent to play the best possible moves?

In the game won by Lee Sedol, AlphaGo was apparently surprised by a brilliant and unexpected move from Lee Sedol. After analysing the logs, the Deep Mind CEO said that AlphaGo had evaluated a 1/10000 probability for that specific move to be played by Lee Sedol.

What I don't understand here is : whatever the probability for a good move to be played, why taking the risk ? Why not instead expecting the opponent to always plays the best moves ?

Of course it's always possible that you miss the best move your opponent could play when playing Monte Carlo to evaluate his possibilities, but here it seems that the move was founded. If AlphaGo knew that its strategy could be counter by such a move, why not choosing another strategy, where the worst case scenario would be less "worse".

It appears that AlphaGo did not rate the move as a best possible move for Lee Sedol, just as one that was within its search space. To put into context the board is 19x19, so a 1 in 10000 chance of a move is much lower than chance of the square being picked at random. That likely makes the move that it "found" not worth exploring much deeper.

It is important to note too that the probabilities assigned to moves are equivalent to AlphaGo's rating for quality of that move - i.e. AlphaGo predicted that this was a bad choice for its opponent. Another way of saying this, is "there is a probability p that this move is the best possible one, and therefore worth investigating further". There is no separate quality rating - AlphaGo does not model "opponent's chance of making a move" separately from "opponent's chance of gaining the highest score from this position if he/she makes that move". There is just one probability covering both those meanings 1

As I understand it, AlphaGo rates the probabilities of all possible moves at each game board state that it considers (starting with the current board), and employs the most search effort for deeper searches on the highest rated ones. I don't know the ratios or how many nodes are visited in a typical search, but expect that a 1 in 10000 rating would not have been explored in much detail if at all.

It is not surprising to see the probability calculation in the system logs, as the logs likely contain the ratings for all legal next moves, as well as ratings for things that didn't actually happen in the game but AlphaGo considered in its deeper searches.

It is also not surprising that AlphaGo failed to rate the move correctly. The neural network is not expected to be a perfect oracle that rates all moves perfectly (if it was, then there would be no need to search). In fact, the opposite could be said to be the case - it is surprising (and of course an amazing feat of engineering) just how good the predictions are, good enough to beat a world-class champion. This is not the same as solving the game though. Go remains "unsolved", even if machines can beat humans, there is an unknown amount of additional room for better and better players - and in the immediate future that could be human or machine.

1. There are in fact two networks evaluating two different things - the "policy network" evaluates potential moves, and the output of that affects the Monte Carlo search. There is also a "value network" which assesses board states to score the end point of the search. It is the policy network that predicted the low probability of the move, which meant that the search had little or no chance of exploring game states past Lee Sedol's move (if it had, maybe the value network would of detected a poor end result from playing that through). In reinforcement learning, a policy is set of rules, based on known state, that decide between actions that an agent can take.
• Nice explanation! Are there any papers/tech articles for a read on how AlphaGo works? I understand that it's Monte-Carlo search with deep learning, but want to know it a bit more clearly! Mar 28, 2016 at 6:43
• Thank you ! So the premises of my question are just totally wrong (not sure if it just comes from me or from some badly written and misleading articles), should I somehow edit the question to make clear that the premises are wrong ? Mar 28, 2016 at 7:44
• @agemO: No, in general it is best to leave the question as-is in this respect, because if you correct it with information from an answer, the question/answer pair won't match up. Mar 28, 2016 at 8:50
• @Dawny33: I haven't read any formal papers on AlphaGo, my answer is extrapolated from sources such as googleresearch.blogspot.co.uk/2016/01/… and from what I know about reinforcement learning combined with neural networks (in fact I am studying this in my spare time this year, so only a beginner). I've not seen code for AlphaGo, but you can find code for variations of DeepMind online e.g. github.com/spragunr/deep_q_rl - these are related in that they also use a (deep) NN to estimate the value of a move in a grid-like environment. Mar 28, 2016 at 9:01
• Just downloaded the nature paper from scihub.io. Not sure whether I can share the link here or not :D (But, the paper helps in making better sense of Neil's answer and the AlphaGo approach)! Mar 28, 2016 at 12:01