0
$\begingroup$

Sorry if my question is kind of dumb, I am very new to this field.

I am trying to create a CNN that plays a variant of chess (for the examples, we'll use chess as it is close enough).

My network , which is a policy network, outputs a vector of planes of probabilities. (e.g 1st pawn layer, 2nd rook layer etc..) each layer contains scalars determining how "good" would a move of the piece to this case be, according to the network. My question is, given the input channels I. And expected outpus (0s everywhere and 1 for the move that was played), how to calculate loss for gradient descent ? (e.g : how to evaluate the "closeness" of a move to another.).

My wild guess is to weight each layer a "wrongness" factor (e.g if instead of wanting to move the first pawn , it tries to move the queen , it is labelled as really "wrong" and then apply some kind of spacial locality (e.g if it doesn't move the first pawn but the second and on the very right case, it is not very wrong). But is it correct ? And in general , how to compute the loss of a non classifying convolutional neural network ?

$\endgroup$
1
$\begingroup$

My wild guess is to weight each layer a "wrongness" factor (e.g if instead of wanting to move the first pawn , it tries to move the queen , it is labelled as really "wrong" and then apply some kind of spacial locality (e.g if it doesn't move the first pawn but the second and on the very right case, it is not very wrong). But is it correct ?

You can relatively simply teach a policy network to predict human-like moves in your scheme using a database of moves. Actually your "wrongness" is probably well-enough represented by classification (your positive class might be "This is a good move") and the usual log loss that goes with it.

After you have trained a policy network, you will want to look in depth at the literature for game-playing bots. Your policy network might work quite well alongside Monte Carlo Tree Search, provided you have some kind of evaluation heuristic for the resulting position.

A reinforcement learning approach to learn from self-play would take you further, enabling the bot to teach itself about good and bad moves/positions, but is too complex to explain in an answer here. I suggest look into the subject after training your network and seeing how good a player you can create using just the policy network and a move search algorithm.

And in general , how to compute the loss of a non classifying convolutional neural network ?

There are a few common options available for regression, such as mean square error (MSE) i.e. $\frac{1}{2N}\sum_{i=1}^{N}(\hat{y}_i - y_i)^2$ where $\hat{y}_i$ is your prediction and $y_i$ is the ground truth for each example. If you use this loss function, and want to predict values outside of range 0-1, remember to use a linear output layer (i.e. no activation function after the last layer), so that the network can actually output close to the values you need - that's about the only difference in network architecture you need to care about.

In the more general case of game-playing bots, it is usual (but not required) to calculate a predicted "return" or "utility" which is the sum of all rewards that will be gained by continuing to act in a certain way. MSE loss is a good choice for that. Although for zero-sum two player games where the reward is simply win/lose, you can use a sigmoid output layer (predicting chance of a win) and cross-entropy loss, much like a classifier.

For your specific case, you can treat your initial policy network as a classifier. This immediately gives you some probability weightings for the predicted move, which you can use to pick the predicted best play, or maybe to guide Monte Carlo Tree Search.

The kind of network that predicts utility or return from a position (or combination of position and action) is called a value network (or an action-value network) in reinforcement learning . If you have both a policy network and a value network, then you are on the way to creating an actor-critic algorithm.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.