1
$\begingroup$

I used a training data set to train both a random forest and a neural network (one hidden layer). Then I compared how both systems perform on a test data set. Interestingly, both turned out to have about the same prediction probabilities. The forest classified 86% of the data correctly into category 0 or 1, the network achieved 85%. As there is 75% of the data in category 0, I am not too happy with the result and hoped for something better.

I then analyzed, which data sets were classified incorrectly. It turned out that there is a large overlap: 86% percent of the data, which was classified incorrectly by the forest, was also classified wrongly by the network. I then compared the probabilities which were attributed to those test samples by the forest and the network. Also the probabilities were almost always comparable so that when the forest was sure that sample A belongs to category 0, then also the network was sure that 0 is the right category. Also, if the probabilities of the forest were around 50% for both categories, also the network was "unsure" which category was correct.

Is such an outcome somehow obvious? Is there an explanation?

$\endgroup$
2
$\begingroup$

I think that your neural network is probably learning the same features as your Random Forest. I do not think it is obvious because it is hard to say what the neural network is learning, but in this case based on the your model it might be working similar.

$\endgroup$
  • $\begingroup$ Is there a way to find out, which features the network learned and considered useful? For the forest, it is easy to get a histogram of the importance of every feature, but for a network I do not know of any method. $\endgroup$ – Merlin1896 Aug 10 '16 at 7:05
  • 2
    $\begingroup$ @Merlin1896: It is likely that the network will have created a bunch of weird mixed-up features in the hidden layer, that you will find hard to interpret. Often neural networks are called "black box" algorithms for the reason that they are hard to interpret. $\endgroup$ – Neil Slater Aug 10 '16 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.