1
$\begingroup$

As far as I understood from the AlphaGo Zero system:

  • During the self-play part, the MCTS algorithm stores a tuple ($s$, $\pi$, $z$) where $s$ is the state, $\pi$ is the distribution probability over the actions in the state and $z$ is an integer representing the winner of the game that state is in.
  • The network will receive $s$ as input (a stack of matrices describing the state $s$) and will output two values: $p$ and $v$. $p$ is a distribution probability over the actions and $v$ is a value in $[-1,1]$ representing which player is likely to win the game.
  • For the training it will use the following loss function:

$$l = (z - v)^2 - \pi^T log(p) + c ||\theta||^2$$

  • Lastly, it evaluates the new network and it starts the self-play section again.

My questions

  • If the network receives only the state $s$ (represented as matrices) as input, how can it then calculate the loss function if the values $\pi$ and $z$ are needed?

  • If these values are indeed passed as input for the network, are they passed through the convolutional (and other) layers of the network? Because if this is true, there is no mention in the article (unless I missed it) of this information.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Hey there, welcome to DS stack exchange! Just a quick suggestion: it may be worth separating your actual questions into separate questions on DSSE (particularly the last one) :) $\endgroup$
    – Dan Carter
    Nov 17, 2019 at 20:27
  • $\begingroup$ Yeah, I figured so, the last one is really different from the other ones. The first two though are closely related. Should I delete the last one? $\endgroup$
    – ihavenoidea
    Nov 17, 2019 at 20:31
  • $\begingroup$ Well to answer that last question: the loss function is applied to the weights in all layers, but the regularisation (l1 or l2) is added to the loss function for each layer individually (stackoverflow.com/questions/53976243/…) $\endgroup$
    – Dan Carter
    Nov 17, 2019 at 20:52

1 Answer 1

Reset to default
1
$\begingroup$

The best way to understand that part is by looking at figure 1 in the AlphaGo Zero paper.

The neural network (NN) minimizes the differences between its own policy $p_t$ and the MCTS policy $\pi_t$. The value of $\pi_t$ is produced by the MCTS self-play which in return uses the NN from the previous iteration.

The same goes for $v_t$ and $z$. In each iteration the weights of the NN are adjusted to minimize the distance between $v_t$ (output of the NN) and $z$ (output of the MCTS) as defined by the loss function. $z$ does not have a time index here as the full self-play produces just a single value for $z$ each time it is conducted.

TLDR for your first question: Both, $\pi$ and $v$, are being produced by the MCTS as input to the NN.

(The indexing in the paper is a bit confusing in my opinion so it is probably easiest to just look at it as stated above)

Now, with "input" I do not mean input on the input layer of the NN. As described in the appendix under "Neural network architecture" the input is a "19 x 19 x 17 image stack". which contains the following information:

  • The positions of player 1 for the latest 8 rounds (8 feature planes)
  • The positions of player 2 for the latest 8 rounds (8 feature planes)
  • A color feature indicating whose turn it is (1 feature planes)

And those 17 feature planes ($8+8+1$) combined with the $19\cdot19$ sized board is the $19\cdot19\cdot17$ input the NN receives thru its input layer. $\pi$ and $z$ are passed to the NN via the loss function only (i.e. they are the target values in this supervised learning problem!).

TLDR for your second question: $\pi$ and $z$ are not fed to the NN thru the input layer but just via the loss function as target values.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you, excellent answer! Just one more side question: do you have any implementation (or an example) that shows how to pass both pi and z directly to the loss function in Keras (with Tensorflow backend)? I am fairly new in Keras. Because as far as I know, we create the Model before the whole AlphaGo Zero loop (MCTS -> NN -> evaluate -> repeat), therefore, I don't know how to pass both pi and z to the NN (for the loss function) if at the model creation they are still not created yet. (Let me know if this question should constitute a whole new question in this site and I will do it). $\endgroup$
    – ihavenoidea
    Nov 18, 2019 at 17:31
  • $\begingroup$ Unfortunately I do not since I have no experience with Keras. Might indeed be a good new question! $\endgroup$
    – Jonathan
    Nov 18, 2019 at 17:37
  • $\begingroup$ @ihavenoidea this might have been unclear but feeding $\pi$ and $z$ to the loss function is actually straight forward. We are in supervised learning so these are the target values. I'll edit my answer to make that clearer. $\endgroup$
    – Jonathan
    Nov 19, 2019 at 8:30
  • $\begingroup$ Yes! However, the problem is that when creating my custom loss function in Keras (the one in the article is not an existing one). So when doing my own loss function, it is passed as parameter only y_true and y_pred, but the network consists of two "y_true" and "y_pred". So I don't know how to exactly get the labels of each head separately. I'll probably open another question regarding this. $\endgroup$
    – ihavenoidea
    Nov 19, 2019 at 16:46
  • $\begingroup$ I don't understand why the latest 8 rounds were kept? What if the repetition happed in the past 9 round? Repetition prevention can be easily achieved by keeping an array of all history shapes of the boards and comparing all of them on each move. When I looked at the 8, I was thinking fo all possible rotations and reflections of the board are 8. $\endgroup$
    – Qian Chen
    Sep 14, 2020 at 2:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.