I have a model that is used in a reinforcement learning algorithm for checkers, a la AlphaZero. Similar to that network, mine features batch normalization after each convolution layer. I am aware that this will cause different behavior/output when using .eval() vs .train()
However, I am unsure of when to use eval() vs train(). The model is used at two different points in the algorithm: First, the network is used to generate many games of self-play. Secondly, the network is trained using the positions of theses games, with the evaluation labels taken from the terminal value of the game (-1, 0, +1) and the 'improved policy' labels are taken to be the visit counts after the UCB-tree-search.
It seems to me that when the network is fully trained, I will use .eval(), as that should be 'what the network really thinks'. Therefore, for the games of self-play, I should also use .eval(). Ostensibly, this should result in stronger games of self-play and thus higher quality data. Finally, if I used .eval() in the self-play step I must also use it in the learning phase, otherwise if the network outputs are different the loss won't even be calculated using the actual outputs! I know that the network learns valuable information from .train(), as the batch norm layers learn about the mean/variance of the data universe.
As I type this, I am starting to suspect that I should be using .train() for both the self-play and learning phases. Still that seems wrong as the discrepancy in output between train() and eval() for a given position can be quite large.