Build neural network to calculate points for a board game

I want to be able to calculate how much points each player has for a board game by taking a picture on the board game. I do this as hobby, not for university or professional purposes. I will use it while playing with my friends and also I hope this will help me build some knowledge about ML.

I plan to do that with supervised learning. Take many pictures of the game and tell the machine what is the correct output. The game scoring is not something straight-forward. Players may control different territories. A territory is controller if a player has units there even if there is enemy buildings. The game is Scythe. From my previous knowledge about ML I know that small changes in the image (for example the angle) from the images you gave to the machine to learn may bug it to recognise them.

Questions (I do not want someone to tell me how to do it but rather which approach is better so I can read more about it):

1. Is supervised learning the best option in this case?
2. I plan to build many micro neural-networks for the different cases - recognise who controls a territory and which territory (which territory seems very hard); separate network to recognise if there are resources on some players territory (this seems very hard, as the combination of resources might be from 0 to many; shall I just put different combinations of resources (+numbers of them) and take many pictures?). There are lots of combinations. I cannot take a pic on every of them.

Theoretically, you could take many pictures and map these pictures to the score of each player. However, I would advise against it. First, you would need plenty of pictures and it might be infeasible to cover all possible game scenarios. Second, game scoring is discrete whereas a traditional neural net would approach it as a regression. This means that your outputs will likely be close but rarely perfectly on the money (and you will need to round up). You could however try to predict who is currently winning and turn this into a classification problem.

Now back to predicting points. Since there already exists a complex point-scoring system based on the board, it would be rather silly to try to let a neural network figure it out on its own. Instead, I would suggest that you implement the point-scoring system and have the neural network translate a picture of the board into a data representation of that board. Your system would do something like this:

def scoring_system(board):
// Insert logic here
return points

def your_neural_network(image):
return board

points = scoring_system(your_neural_network(image))


So, instead of having this be a regression problem, you have multiple problems to solve:

• object detection (where are the pieces)
• object classification (what type of piece is it?)
• image segmentation (where is the board, what territories are there) Etc.

Now some of these bits could be solved with traditional computer vision, but neural networks can be well-equipped as well.

• This is the correct approach. Upvoted. Jun 27, 2020 at 4:12
• Thanks for the comments Valentin and Guillermo. Valentin, this is exactly what I imagine to do. Lets say I want to calculate how much points the black player has from territories. I first ask (I image it like this) one neural network - how much popularity he has. And I know that based on this popularity he gets 3 points per territory. Then I ask another how much territories he has. Its impossible to just do it from one image and it to say player X wins because combinations are so many...
– John
Jun 28, 2020 at 8:44
• Lets talk with a real example - I found a random image on the net: thedailyscythe.com/lets-talk-about-scythe ; Look how the purple player controls 5 territories. I have to detect that. Also one more question - see on the bottom a scoring from 0 to 11. If I want to train a network to recognize that purprple player has 1 score there (as in this case) I imagine to train the network with all kind of combination on the purple player from 0 to 11. Is there a smarter way to do that?
– John
Jun 28, 2020 at 8:55

I think you should use a pre-trained neural network for image recognition and adjust the weights to detect the individual objects you need.

Afterwards, you will need to combine this with some good old-fashioned scripts to manually calculate the score.

Deep Learning doesn't do magic, even less with < 100 pictures of a game. If you managed to take a really huge number of pictures, then maybe we would be having a different discussion!