5
$\begingroup$

I have been trying to replicate the results obtained by AlphaGo following their supervise learning protocol. The papers specify that they use a network that has two heads: a value head that predicts the winner of the game and a policy head that predicts the next move based on the current state of the board.

For the policy head, no problem, the CNN learns to predict the next move. However, the value head does not learn anything and always falls back to always predicting 0 instead of -1/1 for white/black victory. It feels like the model is not complex enough to understand how to predict the outcome of the game so it learns to predict 0 instead to minimize the loss.

Things that I tried and didn't work:

  • Divide the network into two: one value net and one policy net. But even in this case the value net does not learn.

  • Label the winners as 0/1 instead of -1/1 and therefore use a sigmoid instead of a tanh activation function, but that didn't work either.

  • Different number of residual units (3/5/7/11).

  • Only using 2 channels as input, one for black stones and one for white stones.

  • Copy the weights of the first layer of the policy net and freeze them to force the value net to use the already learned features.

Here is the model:

def residual_block(s):

    shortcut = s
    s = Conv2D(256, (3, 3), strides=(1, 1), padding='same', kernel_regularizer=l2(0.0001))(s)
    s = BatchNormalization()(s)
    s = ReLU()(s)
    s = Conv2D(256, (3, 3), strides=(1, 1), padding='same', kernel_regularizer=l2(0.0001))(s)
    s = BatchNormalization()(s)
    s = add([shortcut, s])
    s = ReLU()(s)

    return s

def model():

    inputs = Input(shape=(2, 19, 19, ))

    # Convolutional block
    x = Conv2D(256, (3, 3), strides=(1, 1), padding='same')(inputs)
    x = BatchNormalization()(x)
    x = ReLU()(x)

    # Residual block
    for i in range(11):
        x = residual_block(x)

    # Value head
    x = Conv2D(1, (1, 1), strides=(1, 1), padding='same', kernel_regularizer=l2(0.0001))(x)
    x = BatchNormalization()(x)
    x = ReLU()(x)
    x = Flatten()(x)
    x = Dense(256, kernel_regularizer=l2(0.0001))(x)
    x = ReLU()(x)
    x = Dense(1, activation = 'tanh')(x)

    return Model(inputs=inputs, outputs=x)

And here is the training :

batch_size = 32
gen_train = generator(batch_size, sgf_paths)
gen_test = generator(batch_size, sgf_paths)

optimizer = SGD(lr=0.01, momentum=0.9, decay=0., nesterov=True, clipnorm=1.)
checkp = ModelCheckpoint(filepath="weights_ValueNet.h5", verbose=1, save_best_only=True)

model = model()
model.compile(loss="mean_squared_error", optimizer=optimizer)
hist = model.fit_generator(
        gen_train,
        steps_per_epoch = 2048,
        epochs = 100,
        shuffle = True,
        verbose = 1,
        validation_data = gen_test,
        validation_steps = 300,
        callbacks = [checkp])
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.