How to prevent neural network from overfitting on small subset of features

Question

I'm trying to predict the win probability for a team in a basketball game using a neural network with a single sigmoid output. The input layer consists of a one-hot representation of the players, number of goals(hoops?) for each player, free throws, penalties, etc. For each game I have about 200 training examples each capturing the game state at a particular point in time. My training data spans multiple seasons, so players don't always play for the same team. My hope was that this way I could feed the network a snapshot of the current game state and it would give me the win probability for either team.

Unfortunately, during training the network immediately overfits to the team compositions and memorizes the outcome of that particular game and pays no attention to all the other features. Validation accuracy is abysmal. To prevent the network from fingerprinting the team compositions I tried using dropout on the team inputs only. Whereas in my experience keep_prob around 0.8-0.9 usually yields the best results, here I saw the best results with keep_prob 0.35. This kind of works but it doesn't feel great to basically withhold 65% of player information from the network.

What's the best way to prevent this sort of "cheating" where the network overfits to a small subset of features and looks up memorized results?

Answer 1

I believe that using drop-out is the best solution for such a problem.

I also suggest ensembling methods such as RandomForest instead of a neural network because RandomForest is robust w.r.t overfitting. Also, as you mentioned, your data is not that much complex. So, it would be a good idea to use RandomForest.