Let's say I wanted to train a neural network to teach it the rules in a decision tree, so I generated a dataset by feeding arrays of random numbers into the pre-trained decision tree, and then used the random input arrays as the features in the dataset and the output of the decision tree as the output.

I feed my dataset to the neural network, and start it training. If there are 6 features, logically, a two-layer network can't learn a million-bit decision tree, because there simply isn't enough "information capacity", essentially entropy, in the neural network structure to store such a complex ruleset. With decision trees, calculating entropy is fairly simple. But how would you do it for a neural network?

How would you calculate the minimum size neural network needed to learn a given ruleset?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.