I am quite new to this neural network stuff, so please bear with me :)
TL;DR: I want to train a neural network to predict a scalar score from 32 binary features. Training data is expensive to come by, there is a tradeoff between precision and amount of training samples. Which scenario will likely give me a better result:
Training the network with 100 distinct samples of training data where the output (-1 to 1) is averaged from 100 runs of the same sample, and therefore fairly precise
Training the network with 1000 distinct samples of training data where the output (-1 to 1) is averaged from 10 runs of the same sample, and therefore less precise
Training the network with 10000 distinct samples of training data where the output is just binary (-1 or 1), and therefore very imprecise
More context: I am creating an AI for an imperfect information 4-player card game with 32 cards. I already have implemented a MinMax-based tree search that solves the perfect information version of the game, i.e. this can deliver me the score that is reached at the end of the game, assuming perfect play of all players, for the case that the full card distribution is known to all players. In reality, of course, each player only knows their own hand of cards. For the purposes of the AI I get around this by repeating the perfect information game many times while randomly assigning the unknown cards.
I now want to train a neural network that predicts the win probability that is reached with a given hand of cards (of course, not knowing the cards of the other players). I imagine this would be a value between -1 and 1, where 0 means 50% win probability and 1 means 100% win probability. The input features would be 32 binary values, representing the hand of cards. I want to use my MinMax algorithm to generate the training data for the network.
In a perfect world, I would iterate trough 1 Million random hands of cards and determine a precise win probability for each of them by playing 1 Million randomized perfect information games based on that hand. The reality, however, is that my MinMax algorithm is fairly expensive, and I can't improve it much more. So the total amount of perfect information games I can go through is limited.
Now I am wondering: How do I maximize the effectiveness of my training data generation process? I guess the tradeoff is:
- If I go through many perfect information iterations for each given hand, the win probability in my training data will be fairly close to the 'real' win probability, so very precise
- If I go through fewer (or in extreme case, only 1) perfect information iterations for each given hand, the win probability in my training data will be less precise. However, statistically it should still all even out in the end. Plus, I will have a lot more training samples, covering a much wider range of situations.
In that context I am wondering which side of this spectrum - precision vs. amount - will give me the better tradeoff.
Side note: For my validation data set, of course I will have to determine a fairly precise win probability for at least some samples, where I will probably use more iterations per sample than for the training data.