Let's say I wanted to train a neural network to teach it the rules in a decision tree, so I generated a dataset by feeding arrays of random numbers into the pre-trained decision tree, and then used the random input arrays as the features in the dataset and the output of the decision tree as the output.
I feed my dataset to the neural network, and start it training. If there are 6 features, logically, a two-layer network can't learn a million-bit decision tree, because there simply isn't enough "information capacity", essentially entropy, in the neural network structure to store such a complex ruleset. With decision trees, calculating entropy is fairly simple. But how would you do it for a neural network?
How would you calculate the minimum size neural network needed to learn a given ruleset?